<!DOCTYPE html>
<html lang="zh-CN">
  <head>
    <meta charset="utf-8">
    <meta name="viewport" content="width=device-width,initial-scale=1">
    <title>大数据软件工程总结 | 月藤的博客</title>
    <meta name="generator" content="VuePress 1.8.0">
    <link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/KaTeX/0.7.1/katex.min.css">
    <link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/github-markdown-css/2.10.0/github-markdown.min.css">
    <meta name="description" content="简单的介绍">
    
    <link rel="preload" href="/assets/css/0.styles.a2e941e0.css" as="style"><link rel="preload" href="/assets/js/app.54ea76a2.js" as="script"><link rel="preload" href="/assets/js/2.c554ece5.js" as="script"><link rel="preload" href="/assets/js/3.9d562ae1.js" as="script"><link rel="prefetch" href="/assets/js/10.95d63e51.js"><link rel="prefetch" href="/assets/js/11.26e082be.js"><link rel="prefetch" href="/assets/js/12.1f62021c.js"><link rel="prefetch" href="/assets/js/13.9096527c.js"><link rel="prefetch" href="/assets/js/14.86142c70.js"><link rel="prefetch" href="/assets/js/15.1adeebc5.js"><link rel="prefetch" href="/assets/js/16.efcc79e4.js"><link rel="prefetch" href="/assets/js/17.74da5698.js"><link rel="prefetch" href="/assets/js/18.fd808b3a.js"><link rel="prefetch" href="/assets/js/19.8dafe26f.js"><link rel="prefetch" href="/assets/js/20.aed54ab4.js"><link rel="prefetch" href="/assets/js/21.341a5670.js"><link rel="prefetch" href="/assets/js/22.ab8b375c.js"><link rel="prefetch" href="/assets/js/23.28489470.js"><link rel="prefetch" href="/assets/js/24.87aae001.js"><link rel="prefetch" href="/assets/js/25.211ca3bf.js"><link rel="prefetch" href="/assets/js/26.afa4c8f0.js"><link rel="prefetch" href="/assets/js/27.9b98f6f3.js"><link rel="prefetch" href="/assets/js/28.93298733.js"><link rel="prefetch" href="/assets/js/29.e8c038c4.js"><link rel="prefetch" href="/assets/js/30.27939f01.js"><link rel="prefetch" href="/assets/js/31.6a9774e8.js"><link rel="prefetch" href="/assets/js/32.f698e967.js"><link rel="prefetch" href="/assets/js/33.5ba4bfb8.js"><link rel="prefetch" href="/assets/js/34.dca90861.js"><link rel="prefetch" href="/assets/js/35.768c1bb2.js"><link rel="prefetch" href="/assets/js/36.6b0a867f.js"><link rel="prefetch" href="/assets/js/37.61156e90.js"><link rel="prefetch" href="/assets/js/38.e390246c.js"><link rel="prefetch" href="/assets/js/39.0fbd0dd1.js"><link rel="prefetch" href="/assets/js/4.e18bc6a8.js"><link rel="prefetch" href="/assets/js/40.4252699c.js"><link rel="prefetch" href="/assets/js/41.2e5b5840.js"><link rel="prefetch" href="/assets/js/42.8fe886c7.js"><link rel="prefetch" href="/assets/js/43.7a4b0851.js"><link rel="prefetch" href="/assets/js/44.93c2313d.js"><link rel="prefetch" href="/assets/js/45.33dcea60.js"><link rel="prefetch" href="/assets/js/46.681fdf10.js"><link rel="prefetch" href="/assets/js/47.a842a141.js"><link rel="prefetch" href="/assets/js/48.9a03ba74.js"><link rel="prefetch" href="/assets/js/49.a50266b1.js"><link rel="prefetch" href="/assets/js/5.dc6a2d8c.js"><link rel="prefetch" href="/assets/js/50.f2a42406.js"><link rel="prefetch" href="/assets/js/51.444cc3d8.js"><link rel="prefetch" href="/assets/js/52.cf2befd8.js"><link rel="prefetch" href="/assets/js/53.fbe609f8.js"><link rel="prefetch" href="/assets/js/54.a43fb514.js"><link rel="prefetch" href="/assets/js/55.d0c11641.js"><link rel="prefetch" href="/assets/js/56.c6f114d9.js"><link rel="prefetch" href="/assets/js/57.cc386420.js"><link rel="prefetch" href="/assets/js/58.e747d0f6.js"><link rel="prefetch" href="/assets/js/6.c1fd48f0.js"><link rel="prefetch" href="/assets/js/7.32b39b92.js"><link rel="prefetch" href="/assets/js/8.e2284671.js"><link rel="prefetch" href="/assets/js/9.8def3992.js">
    <link rel="stylesheet" href="/assets/css/0.styles.a2e941e0.css">
  </head>
  <body>
    <div id="app" data-server-rendered="true"><div class="theme-container"><header class="navbar"><div class="sidebar-button"><svg xmlns="http://www.w3.org/2000/svg" aria-hidden="true" role="img" viewBox="0 0 448 512" class="icon"><path fill="currentColor" d="M436 124H12c-6.627 0-12-5.373-12-12V80c0-6.627 5.373-12 12-12h424c6.627 0 12 5.373 12 12v32c0 6.627-5.373 12-12 12zm0 160H12c-6.627 0-12-5.373-12-12v-32c0-6.627 5.373-12 12-12h424c6.627 0 12 5.373 12 12v32c0 6.627-5.373 12-12 12zm0 160H12c-6.627 0-12-5.373-12-12v-32c0-6.627 5.373-12 12-12h424c6.627 0 12 5.373 12 12v32c0 6.627-5.373 12-12 12z"></path></svg></div> <a href="/" class="home-link router-link-active"><!----> <span class="site-name">月藤的博客</span></a> <div class="links"><div class="search-box"><input aria-label="Search" autocomplete="off" spellcheck="false" value=""> <!----></div> <nav class="nav-links can-hide"><div class="nav-item"><a href="https://ziphold.gitee.io/blog/#/" target="_blank" rel="noopener noreferrer" class="nav-link external">
  旧博客（停止更新）
  <span><svg xmlns="http://www.w3.org/2000/svg" aria-hidden="true" focusable="false" x="0px" y="0px" viewBox="0 0 100 100" width="15" height="15" class="icon outbound"><path fill="currentColor" d="M18.8,85.1h56l0,0c2.2,0,4-1.8,4-4v-32h-8v28h-48v-48h28v-8h-32l0,0c-2.2,0-4,1.8-4,4v56C14.8,83.3,16.6,85.1,18.8,85.1z"></path> <polygon fill="currentColor" points="45.7,48.7 51.3,54.3 77.2,28.5 77.2,37.2 85.2,37.2 85.2,14.9 62.8,14.9 62.8,22.9 71.5,22.9"></polygon></svg> <span class="sr-only">(opens new window)</span></span></a></div><div class="nav-item"><a href="https://rattonlzh.github.io/homepage/homepage.html" target="_blank" rel="noopener noreferrer" class="nav-link external">
  常用网址导航（停止更新）
  <span><svg xmlns="http://www.w3.org/2000/svg" aria-hidden="true" focusable="false" x="0px" y="0px" viewBox="0 0 100 100" width="15" height="15" class="icon outbound"><path fill="currentColor" d="M18.8,85.1h56l0,0c2.2,0,4-1.8,4-4v-32h-8v28h-48v-48h28v-8h-32l0,0c-2.2,0-4,1.8-4,4v56C14.8,83.3,16.6,85.1,18.8,85.1z"></path> <polygon fill="currentColor" points="45.7,48.7 51.3,54.3 77.2,28.5 77.2,37.2 85.2,37.2 85.2,14.9 62.8,14.9 62.8,22.9 71.5,22.9"></polygon></svg> <span class="sr-only">(opens new window)</span></span></a></div><div class="nav-item"><div class="dropdown-wrapper"><button type="button" aria-label="Select language" class="dropdown-title"><span class="title">Languages</span> <span class="arrow down"></span></button> <button type="button" aria-label="Select language" class="mobile-dropdown-title"><span class="title">Languages</span> <span class="arrow right"></span></button> <ul class="nav-dropdown" style="display:none;"><li class="dropdown-item"><!----> <a href="/知识点总结/大数据软件工程总结.html" class="nav-link">
  zh-CN
</a></li><li class="dropdown-item"><!----> <a href="/en/" class="nav-link">
  en-US
</a></li></ul></div></div> <!----></nav></div></header> <div class="sidebar-mask"></div> <aside class="sidebar"><div class="avatar"><img src="/assets/img/avatar.2b77755b.png" alt srcset></div> <div style="z-index: 999"><iframe frameborder="no" border="0" marginwidth="0" marginheight="0" width="298" height="52" src="//music.163.com/outchain/player?type=2&id=28283137&auto=1&height=32"></iframe></div> <nav class="nav-links"><div class="nav-item"><a href="https://ziphold.gitee.io/blog/#/" target="_blank" rel="noopener noreferrer" class="nav-link external">
  旧博客（停止更新）
  <span><svg xmlns="http://www.w3.org/2000/svg" aria-hidden="true" focusable="false" x="0px" y="0px" viewBox="0 0 100 100" width="15" height="15" class="icon outbound"><path fill="currentColor" d="M18.8,85.1h56l0,0c2.2,0,4-1.8,4-4v-32h-8v28h-48v-48h28v-8h-32l0,0c-2.2,0-4,1.8-4,4v56C14.8,83.3,16.6,85.1,18.8,85.1z"></path> <polygon fill="currentColor" points="45.7,48.7 51.3,54.3 77.2,28.5 77.2,37.2 85.2,37.2 85.2,14.9 62.8,14.9 62.8,22.9 71.5,22.9"></polygon></svg> <span class="sr-only">(opens new window)</span></span></a></div><div class="nav-item"><a href="https://rattonlzh.github.io/homepage/homepage.html" target="_blank" rel="noopener noreferrer" class="nav-link external">
  常用网址导航（停止更新）
  <span><svg xmlns="http://www.w3.org/2000/svg" aria-hidden="true" focusable="false" x="0px" y="0px" viewBox="0 0 100 100" width="15" height="15" class="icon outbound"><path fill="currentColor" d="M18.8,85.1h56l0,0c2.2,0,4-1.8,4-4v-32h-8v28h-48v-48h28v-8h-32l0,0c-2.2,0-4,1.8-4,4v56C14.8,83.3,16.6,85.1,18.8,85.1z"></path> <polygon fill="currentColor" points="45.7,48.7 51.3,54.3 77.2,28.5 77.2,37.2 85.2,37.2 85.2,14.9 62.8,14.9 62.8,22.9 71.5,22.9"></polygon></svg> <span class="sr-only">(opens new window)</span></span></a></div><div class="nav-item"><div class="dropdown-wrapper"><button type="button" aria-label="Select language" class="dropdown-title"><span class="title">Languages</span> <span class="arrow down"></span></button> <button type="button" aria-label="Select language" class="mobile-dropdown-title"><span class="title">Languages</span> <span class="arrow right"></span></button> <ul class="nav-dropdown" style="display:none;"><li class="dropdown-item"><!----> <a href="/知识点总结/大数据软件工程总结.html" class="nav-link">
  zh-CN
</a></li><li class="dropdown-item"><!----> <a href="/en/" class="nav-link">
  en-US
</a></li></ul></div></div> <!----></nav>  <ul class="sidebar-links"><li><section class="sidebar-group collapsable depth-0"><p class="sidebar-heading"><span>随便说说</span> <span class="arrow right"></span></p> <!----></section></li><li><section class="sidebar-group collapsable depth-0"><p class="sidebar-heading open"><span>知识点总结</span> <span class="arrow down"></span></p> <ul class="sidebar-links sidebar-group-items"><li><a href="/知识点总结/git总结.html" class="sidebar-link">git总结</a></li><li><a href="/知识点总结/java单元测试总结.html" class="sidebar-link">java测试</a></li><li><a href="/知识点总结/jquery总结.html" class="sidebar-link">jquery初探</a></li><li><a href="/知识点总结/js总结.html" class="sidebar-link">js易错点</a></li><li><a href="/知识点总结/kotlin语法总结.html" class="sidebar-link">kotlin语法入门</a></li><li><a href="/知识点总结/latex总结.html" class="sidebar-link">latex常用代码</a></li><li><a href="/知识点总结/PS总结.html" class="sidebar-link">PS总结</a></li><li><a href="/知识点总结/springboot总结.html" class="sidebar-link">springboot进行web开发总结</a></li><li><a href="/知识点总结/thinkphp总结.html" class="sidebar-link">thinkphp入坑</a></li><li><a href="/知识点总结/uml软件建模总结.html" class="sidebar-link">uml软件建模总结</a></li><li><a href="/知识点总结/人工智能总结.html" class="sidebar-link">人工智能总结</a></li><li><a href="/知识点总结/大数据软件工程总结.html" class="active sidebar-link">大数据软件工程总结</a><ul class="sidebar-sub-headers"><li class="sidebar-sub-header"><a href="/知识点总结/大数据软件工程总结.html#概论" class="sidebar-link">概论</a></li><li class="sidebar-sub-header"><a href="/知识点总结/大数据软件工程总结.html#大数据生态系统" class="sidebar-link">大数据生态系统</a></li><li class="sidebar-sub-header"><a href="/知识点总结/大数据软件工程总结.html#大数据采集与预处理" class="sidebar-link">大数据采集与预处理</a></li><li class="sidebar-sub-header"><a href="/知识点总结/大数据软件工程总结.html#大数据库" class="sidebar-link">大数据库</a></li><li class="sidebar-sub-header"><a href="/知识点总结/大数据软件工程总结.html#在pregel计算过程中-一个算法什么时候可以结束-是由所有顶点的状态决定的。" class="sidebar-link">在Pregel计算过程中，一个算法什么时候可以结束，是由所有顶点的状态决定的。</a></li><li class="sidebar-sub-header"><a href="/知识点总结/大数据软件工程总结.html#大数据分析" class="sidebar-link">大数据分析</a></li><li class="sidebar-sub-header"><a href="/知识点总结/大数据软件工程总结.html#大数据可视化技术" class="sidebar-link">大数据可视化技术</a></li><li class="sidebar-sub-header"><a href="/知识点总结/大数据软件工程总结.html#大数据应用" class="sidebar-link">大数据应用</a></li><li class="sidebar-sub-header"><a href="/知识点总结/大数据软件工程总结.html#大数据安全" class="sidebar-link">大数据安全</a></li><li class="sidebar-sub-header"><a href="/知识点总结/大数据软件工程总结.html#参考文献" class="sidebar-link">参考文献</a></li></ul></li><li><a href="/知识点总结/常用快捷键总结.html" class="sidebar-link">第一篇博客</a></li><li><a href="/知识点总结/数字图像处理总结.html" class="sidebar-link">数字图像处理知识点</a></li><li><a href="/知识点总结/数据库应用理论总结.html" class="sidebar-link">数据库应用复习</a></li><li><a href="/知识点总结/数据库系统概念总结.html" class="sidebar-link">数据库系统概念总结</a></li><li><a href="/知识点总结/数据挖掘总结.html" class="sidebar-link">数据挖掘总结</a></li><li><a href="/知识点总结/设计模式总结.html" class="sidebar-link">设计模式</a></li></ul></section></li><li><section class="sidebar-group collapsable depth-0"><p class="sidebar-heading"><span>考研专区</span> <span class="arrow right"></span></p> <!----></section></li><li><section class="sidebar-group collapsable depth-0"><p class="sidebar-heading"><span>读书笔记</span> <span class="arrow right"></span></p> <!----></section></li><li><section class="sidebar-group collapsable depth-0"><p class="sidebar-heading"><span>开发资料</span> <span class="arrow right"></span></p> <!----></section></li><li><section class="sidebar-group collapsable depth-0"><p class="sidebar-heading"><span>配置记录</span> <span class="arrow right"></span></p> <!----></section></li><li><section class="sidebar-group collapsable depth-0"><p class="sidebar-heading"><span>思维导图</span> <span class="arrow right"></span></p> <!----></section></li><li><a href="/friend_link.html" class="sidebar-link">友情链接</a></li></ul> </aside> <main class="page"> <div class="theme-default-content content__default"><h1 id="大数据软件工程总结"><a href="#大数据软件工程总结" class="header-anchor">#</a> 大数据软件工程总结</h1> <p>文中图片基本来自网络或教材ppt</p> <h2 id="概论"><a href="#概论" class="header-anchor">#</a> 概论</h2> <h3 id="_1-了解大数据技术产生的历史必然"><a href="#_1-了解大数据技术产生的历史必然" class="header-anchor">#</a> （1）了解大数据技术产生的历史必然；</h3> <ul><li>数据产生方式的变革促成大数据时代的来临（运营式系统阶段-&gt;用户原创阶段-&gt;感知式系统阶段）</li> <li>云计算是大数据产生的前提和必要条件</li></ul> <h3 id="_2-理解大数据的特征"><a href="#_2-理解大数据的特征" class="header-anchor">#</a> （2）理解大数据的特征；</h3> <p>大数据指在一定时间范围内无法用常规工具捕捉和处理的数据集合</p> <p>具有4V特性：Velocity（实时处理），Variety（多数据类型交叉分析），Volume（PB级数据处理），Value（海量数据挖掘）</p> <h3 id="_3-理解大数据生命周期"><a href="#_3-理解大数据生命周期" class="header-anchor">#</a> （3）理解大数据生命周期；</h3> <p>采集-&gt;存储-&gt;分析-&gt;解释-&gt;应用</p> <p><img src="/assets/img/image-20200626102130600.08da46f6.png" alt="image-20200626102130600"></p> <p><img src="/assets/img/image-20200626102311526.a0be0dad.png" alt="image-20200626102311526"></p> <h3 id="_4-理解大数据、云计算、物联网之间的相互关系"><a href="#_4-理解大数据、云计算、物联网之间的相互关系" class="header-anchor">#</a> （4）理解大数据、云计算、物联网之间的相互关系；</h3> <ul><li>大数据为物联网提供数据分析支撑，物联网是大数据重要的数据来源</li> <li>物联网为云计算提供应用空间，云计算为物联网提供海量存储能力</li> <li>云计算是大数据的技术基础，大数据体现了云计算的应用价值</li></ul> <p><img src="/assets/img/image-20200626103511334.e193bf40.png" alt="image-20200626103511334"></p> <h3 id="_5-了解大数据的变革"><a href="#_5-了解大数据的变革" class="header-anchor">#</a> （5）了解大数据的变革</h3> <p>有8个重要变革</p> <ol><li>决策方式：目标驱动-&gt;数据驱动</li> <li>方法论：基于知识-&gt;基于数据</li> <li>计算方式：复杂-&gt;简单</li> <li>管理方式：业务数据化-&gt;数据业务化</li> <li>研究范式：第三范式-&gt;第四范式</li> <li>数据的属性：数据是资源-&gt;数据是资产</li> <li>数据处理模式：小众参与-&gt;大众协同</li> <li>思维方式：整体思维+相关思维+容错思维</li></ol> <h2 id="大数据生态系统"><a href="#大数据生态系统" class="header-anchor">#</a> 大数据生态系统</h2> <h3 id="_1-掌握hdfs分布式文件系统的结构"><a href="#_1-掌握hdfs分布式文件系统的结构" class="header-anchor">#</a> （1）掌握HDFS分布式文件系统的结构；</h3> <p><img src="/assets/img/image-20200626114046238.22f1de4c.png" alt="image-20200626114046238"></p> <ol><li>HDFS=1个NameNode+1个Secondary NameNode+多个DataNode；</li> <li>NameNode称为名称节点、命名空间、主节点、元数据节点，主要存放元数据 （meta）；账本。</li> <li>Secondary NameNode称为从元数据节点，是命名空间的冷备份（Namenode秘 书），高可用（HA）解决方案，HDFS2.0采用另外一种机制。</li> <li>DataNode称为数据节点，是存放数据的物理空间，以块（block）为基本单位；</li> <li>NameNode和DataNode是主从结构；</li> <li>块是HDFS操作最小单位，一般为128M；</li> <li>元数据(meta)=文件目录结构信息(catalogue)+操作日志信息(log)，数据的描述信息</li> <li>NameNode只存放catalogue，和Secondary NameNode之间只相差3秒的catalogue；</li> <li>DataNode按机架（rack）进行组织，如上图有两个机架；</li> <li>客户端只能同NameNode交互；</li></ol> <h3 id="_2-掌握hdfs存储原理"><a href="#_2-掌握hdfs存储原理" class="header-anchor">#</a> （2）掌握HDFS存储原理；</h3> <p>读数据过程如下：</p> <ol><li>访问NameNode</li> <li>返回MetaData</li> <li>访问DataNode</li> <li>读数据</li></ol> <p>写数据策略</p> <ul><li><p>第一个副本放置在上传文件的数据节点；如果是集群外提交，则随机挑选一 台磁盘不太满、CPU不太忙的节点。</p></li> <li><p>第二个副本放置在与第一个副本不同的机架的节点上。</p></li> <li><p>第三个副本与第一个副本相同机架的其他节点上。</p></li> <li><p>更多副本随机节点</p></li></ul> <h3 id="_3-掌握hdfs读写操作"><a href="#_3-掌握hdfs读写操作" class="header-anchor">#</a> （3）掌握HDFS读写操作；</h3> <p>读操作</p> <ol><li>打开文件</li> <li>获取数据块信息</li> <li>发出读取请求</li> <li>读取数据</li> <li>获取数据块信息（可能发生）</li> <li>读取数据（可能发生）</li> <li>关闭 文件</li></ol> <p>实验代码如下：</p> <div class="language-java extra-class"><pre class="language-java"><code><span class="token keyword">import</span> <span class="token namespace">java<span class="token punctuation">.</span>io<span class="token punctuation">.</span></span><span class="token class-name">BufferedReader</span><span class="token punctuation">;</span>
<span class="token keyword">import</span> <span class="token namespace">java<span class="token punctuation">.</span>io<span class="token punctuation">.</span></span><span class="token class-name">InputStreamReader</span><span class="token punctuation">;</span>
<span class="token keyword">import</span> <span class="token namespace">org<span class="token punctuation">.</span>apache<span class="token punctuation">.</span>hadoop<span class="token punctuation">.</span>conf<span class="token punctuation">.</span></span><span class="token class-name">Configuration</span><span class="token punctuation">;</span>
<span class="token keyword">import</span> <span class="token namespace">org<span class="token punctuation">.</span>apache<span class="token punctuation">.</span>hadoop<span class="token punctuation">.</span>fs<span class="token punctuation">.</span></span><span class="token class-name">FileSystem</span><span class="token punctuation">;</span>
<span class="token keyword">import</span> <span class="token namespace">org<span class="token punctuation">.</span>apache<span class="token punctuation">.</span>hadoop<span class="token punctuation">.</span>fs<span class="token punctuation">.</span></span><span class="token class-name">Path</span><span class="token punctuation">;</span>
<span class="token keyword">import</span> <span class="token namespace">org<span class="token punctuation">.</span>apache<span class="token punctuation">.</span>hadoop<span class="token punctuation">.</span>fs<span class="token punctuation">.</span></span><span class="token class-name">FSDataInputStream</span><span class="token punctuation">;</span>
<span class="token keyword">public</span> <span class="token keyword">class</span> <span class="token class-name">Chapter3</span> <span class="token punctuation">{</span>
 <span class="token keyword">public</span> <span class="token keyword">static</span> <span class="token keyword">void</span> <span class="token function">main</span><span class="token punctuation">(</span><span class="token class-name">String</span><span class="token punctuation">[</span><span class="token punctuation">]</span> args<span class="token punctuation">)</span> <span class="token punctuation">{</span>
 <span class="token keyword">try</span> <span class="token punctuation">{</span>
 <span class="token class-name">Configuration</span> conf <span class="token operator">=</span> <span class="token keyword">new</span> <span class="token class-name">Configuration</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
 conf<span class="token punctuation">.</span><span class="token function">set</span><span class="token punctuation">(</span><span class="token string">&quot;fs.defaultFS&quot;</span><span class="token punctuation">,</span><span class="token string">&quot;hdfs://localhost:9000&quot;</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
conf<span class="token punctuation">.</span><span class="token function">set</span><span class="token punctuation">(</span><span class="token string">&quot;fs.hdfs.impl&quot;</span><span class="token punctuation">,</span><span class="token string">&quot;org.apache.hadoop.hdfs.DistributedFileSystem&quot;</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
 <span class="token class-name">FileSystem</span> fs <span class="token operator">=</span> <span class="token class-name">FileSystem</span><span class="token punctuation">.</span><span class="token function">get</span><span class="token punctuation">(</span>conf<span class="token punctuation">)</span><span class="token punctuation">;</span>
 <span class="token class-name">Path</span> file <span class="token operator">=</span> <span class="token keyword">new</span> <span class="token class-name">Path</span><span class="token punctuation">(</span><span class="token string">&quot;test&quot;</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
 <span class="token class-name">FSDataInputStream</span> getIt <span class="token operator">=</span> fs<span class="token punctuation">.</span><span class="token keyword">open</span><span class="token punctuation">(</span>file<span class="token punctuation">)</span><span class="token punctuation">;</span>
 <span class="token class-name">BufferedReader</span> d <span class="token operator">=</span> <span class="token keyword">new</span> <span class="token class-name">BufferedReader</span><span class="token punctuation">(</span><span class="token keyword">new</span> <span class="token class-name">InputStreamReader</span><span class="token punctuation">(</span>getIt<span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
 <span class="token class-name">String</span> content <span class="token operator">=</span> d<span class="token punctuation">.</span><span class="token function">readLine</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token comment">//读取文件一行</span>
 <span class="token class-name">System</span><span class="token punctuation">.</span>out<span class="token punctuation">.</span><span class="token function">println</span><span class="token punctuation">(</span>content<span class="token punctuation">)</span><span class="token punctuation">;</span>
 d<span class="token punctuation">.</span><span class="token function">close</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token comment">//关闭文件</span>
 fs<span class="token punctuation">.</span><span class="token function">close</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token comment">//关闭hdfs</span>
 <span class="token punctuation">}</span> <span class="token keyword">catch</span> <span class="token punctuation">(</span><span class="token class-name">Exception</span> e<span class="token punctuation">)</span> <span class="token punctuation">{</span> e<span class="token punctuation">.</span><span class="token function">printStackTrace</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token punctuation">}</span>
 <span class="token punctuation">}</span><span class="token punctuation">}</span><span class="token punctuation">}</span>
</code></pre></div><p>写操作</p> <ol><li>创建文件请求</li> <li>创建文件元数据</li> <li>写入数据</li> <li>写入数据包</li> <li>接受确认包</li> <li>关闭文件</li> <li>写操作完成</li></ol> <p>实验代码如下：</p> <div class="language-java extra-class"><pre class="language-java"><code><span class="token keyword">import</span> <span class="token namespace">org<span class="token punctuation">.</span>apache<span class="token punctuation">.</span>hadoop<span class="token punctuation">.</span>conf<span class="token punctuation">.</span></span><span class="token class-name">Configuration</span><span class="token punctuation">;</span>
<span class="token keyword">import</span> <span class="token namespace">org<span class="token punctuation">.</span>apache<span class="token punctuation">.</span>hadoop<span class="token punctuation">.</span>fs<span class="token punctuation">.</span></span><span class="token class-name">FileSystem</span><span class="token punctuation">;</span>
<span class="token keyword">import</span> <span class="token namespace">org<span class="token punctuation">.</span>apache<span class="token punctuation">.</span>hadoop<span class="token punctuation">.</span>fs<span class="token punctuation">.</span></span><span class="token class-name">FSDataOutputStream</span><span class="token punctuation">;</span>
<span class="token keyword">import</span> <span class="token namespace">org<span class="token punctuation">.</span>apache<span class="token punctuation">.</span>hadoop<span class="token punctuation">.</span>fs<span class="token punctuation">.</span></span><span class="token class-name">Path</span><span class="token punctuation">;</span>
<span class="token keyword">public</span> <span class="token keyword">class</span> <span class="token class-name">Chapter3</span> <span class="token punctuation">{</span>
 <span class="token keyword">public</span> <span class="token keyword">static</span> <span class="token keyword">void</span> <span class="token function">main</span><span class="token punctuation">(</span><span class="token class-name">String</span><span class="token punctuation">[</span><span class="token punctuation">]</span> args<span class="token punctuation">)</span> <span class="token punctuation">{</span>
 <span class="token keyword">try</span> <span class="token punctuation">{</span>
 <span class="token class-name">Configuration</span> conf <span class="token operator">=</span> <span class="token keyword">new</span> <span class="token class-name">Configuration</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
conf<span class="token punctuation">.</span><span class="token function">set</span><span class="token punctuation">(</span><span class="token string">&quot;fs.defaultFS&quot;</span><span class="token punctuation">,</span><span class="token string">&quot;hdfs://localhost:9000&quot;</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
conf<span class="token punctuation">.</span><span class="token function">set</span><span class="token punctuation">(</span><span class="token string">&quot;fs.hdfs.impl&quot;</span><span class="token punctuation">,</span><span class="token string">&quot;org.apache.hadoop.hdfs.DistributedFileSystem&quot;</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
 <span class="token class-name">FileSystem</span> fs <span class="token operator">=</span> <span class="token class-name">FileSystem</span><span class="token punctuation">.</span><span class="token function">get</span><span class="token punctuation">(</span>conf<span class="token punctuation">)</span><span class="token punctuation">;</span>
 <span class="token keyword">byte</span><span class="token punctuation">[</span><span class="token punctuation">]</span> buff <span class="token operator">=</span> <span class="token string">&quot;Hello world&quot;</span><span class="token punctuation">.</span><span class="token function">getBytes</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token comment">// 要写入的内容</span>
 <span class="token class-name">String</span> filename <span class="token operator">=</span> <span class="token string">&quot;test&quot;</span><span class="token punctuation">;</span> <span class="token comment">//要写入的文件名</span>
 <span class="token class-name">FSDataOutputStream</span> os <span class="token operator">=</span> fs<span class="token punctuation">.</span><span class="token function">create</span><span class="token punctuation">(</span><span class="token keyword">new</span> <span class="token class-name">Path</span><span class="token punctuation">(</span>filename<span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
 os<span class="token punctuation">.</span><span class="token function">write</span><span class="token punctuation">(</span>buff<span class="token punctuation">,</span><span class="token number">0</span><span class="token punctuation">,</span>buff<span class="token punctuation">.</span>length<span class="token punctuation">)</span><span class="token punctuation">;</span>
 <span class="token class-name">System</span><span class="token punctuation">.</span>out<span class="token punctuation">.</span><span class="token function">println</span><span class="token punctuation">(</span><span class="token string">&quot;Create:&quot;</span><span class="token operator">+</span> filename<span class="token punctuation">)</span><span class="token punctuation">;</span>
 os<span class="token punctuation">.</span><span class="token function">close</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
 fs<span class="token punctuation">.</span><span class="token function">close</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
 <span class="token punctuation">}</span> <span class="token keyword">catch</span> <span class="token punctuation">(</span><span class="token class-name">Exception</span> e<span class="token punctuation">)</span> <span class="token punctuation">{</span> e<span class="token punctuation">.</span><span class="token function">printStackTrace</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token punctuation">}</span> 
</code></pre></div><h3 id="_4-掌握mapreduce编程框架"><a href="#_4-掌握mapreduce编程框架" class="header-anchor">#</a> （4）掌握MapReduce编程框架</h3> <p><img src="/assets/img/image-20200626120138593.52d8dfce.png" alt="image-20200626120138593"></p> <ol><li><p><strong>Client</strong>：用户编写的MapReduce程序通过Client提交到JobTracker</p></li> <li><p><strong>JobTracker</strong>:</p> <ul><li>负责资源监控和作业调度；</li> <li>监控TaskTracker与Job的健康情况，如果发现失败，则将任务转移到其他节点</li> <li>跟踪资源使用量、执行进度信息，报告给TaskScheduler</li></ul></li> <li><p><strong>TaskTracker</strong>：</p> <ul><li>周期地通过“心跳”将使用情况和执行进度报告给Job Tracker</li> <li>接受Job Tracker发出的命令并执行操作</li> <li>将cpu，内存等资源划分成“slot”，分为 map slot和reduce slot两种。一个task获得了slot才能运行</li></ul></li> <li><p><strong>Task</strong>:分为Map Task和Reduce Task，由Task Tracker启动</p></li> <li><p><strong>Task Scheduler</strong>：将各个Task Tracker的空闲slot分配给task使用</p></li></ol> <h3 id="_5-理解map原理"><a href="#_5-理解map原理" class="header-anchor">#</a> （5）理解Map原理；</h3> <ol><li><p>将小批数据集进一步解析成一批<code>&lt;key, value&gt;</code>对，输入map函数进行处理</p></li> <li><p>每一个 <code>&lt;key, value&gt;</code>会输出一批 <code>&lt;k2,v2&gt;</code>，这是计算的中间结果，用作reduce函数的输入，reduce函数输出最终结果</p></li></ol> <h3 id="_6-理解mapreduce原理"><a href="#_6-理解mapreduce原理" class="header-anchor">#</a> （6）理解MapReduce原理；</h3> <p>所有的数据交换都是通过MapReduce框架自身取完成</p> <p><img src="/assets/img/image-20200626124630337.7db83545.png" alt="image-20200626124630337"></p> <ol><li><p>从分布式文件系统加载文件</p> <ol><li>根据自定义的规则将数据划分成多个split（逻辑的划分），理想的分片大小是一个hdfs块</li></ol></li> <li><p>每个split创建一个map任务，将数据处理成一批键值对</p></li> <li><p>进行shuffle</p> <p>map的shuffle</p> <ol><li>Map任务的输出写入缓存</li> <li>进行溢写（分区、排序、合并）</li> <li>文件归并</li></ol> <p>（合并（Combine）和归并（Merge）的区别： 两个键值对&lt;“a”,1&gt;和&lt;“a”,1&gt;，如果合并，会得到&lt;“a”,2&gt;，如果归并，会得到&lt;“a”,&lt;1,1&gt;&gt;）</p> <p>reduce的shuffle</p> <ol><li>向JobTracker询问map任务是否完成，若完成，则领取数据</li> <li>数据放入缓存，来自不同map任务的，先归并，再合并，写入磁盘（如果数据很少，则不写入磁盘，直接在缓存中归并，输出给reduce，跳过3</li> <li>多个溢写文件归并成一个或多个大文件，文件的键值是排序的</li></ol></li> <li><p>中间结果输入到reduce函数，处理得到最终结果，最终结果写入到分布式文件系统（如HDFS）。reduce任务的个数一般设成比reduce任务槽（slot）数目稍小</p></li></ol> <h3 id="demo-wordcount程序"><a href="#demo-wordcount程序" class="header-anchor">#</a> demo：wordcount程序</h3> <p>mapper</p> <div class="language-java extra-class"><pre class="language-java"><code><span class="token keyword">public</span> <span class="token keyword">class</span> <span class="token class-name">WordCountMapper</span> <span class="token keyword">extends</span> <span class="token class-name">Mapper</span><span class="token generics"><span class="token punctuation">&lt;</span><span class="token class-name">LongWritable</span><span class="token punctuation">,</span><span class="token class-name">Text</span><span class="token punctuation">,</span><span class="token class-name">Text</span><span class="token punctuation">,</span><span class="token class-name">IntWritable</span><span class="token punctuation">&gt;</span></span> <span class="token punctuation">{</span>
 <span class="token comment">//对数据进行打散</span>
 <span class="token annotation punctuation">@Override</span>
 <span class="token keyword">protected</span> <span class="token keyword">void</span> <span class="token function">map</span><span class="token punctuation">(</span><span class="token class-name">LongWritable</span> key<span class="token punctuation">,</span> <span class="token class-name">Text</span> value<span class="token punctuation">,</span> <span class="token class-name">Context</span> context<span class="token punctuation">)</span> <span class="token keyword">throws</span>
<span class="token class-name">IOException</span><span class="token punctuation">,</span> <span class="token class-name">InterruptedException</span> <span class="token punctuation">{</span>
 <span class="token comment">//1.接入数据 hello reba hello mimi</span>
 <span class="token class-name">String</span> line <span class="token operator">=</span> value<span class="token punctuation">.</span><span class="token function">toString</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
 <span class="token comment">//2.对数据进行切分</span>
 <span class="token class-name">String</span><span class="token punctuation">[</span><span class="token punctuation">]</span> words <span class="token operator">=</span> line<span class="token punctuation">.</span><span class="token function">split</span><span class="token punctuation">(</span><span class="token string">&quot; &quot;</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
 <span class="token comment">//3.写出以&lt;hello,1&gt;</span>
 <span class="token keyword">for</span><span class="token punctuation">(</span><span class="token class-name">String</span> w<span class="token operator">:</span>words<span class="token punctuation">)</span><span class="token punctuation">{</span>
 <span class="token comment">//写出reducer端</span>
 context<span class="token punctuation">.</span><span class="token function">write</span><span class="token punctuation">(</span><span class="token keyword">new</span> <span class="token class-name">Text</span><span class="token punctuation">(</span>w<span class="token punctuation">)</span><span class="token punctuation">,</span><span class="token keyword">new</span> <span class="token class-name">IntWritable</span><span class="token punctuation">(</span><span class="token number">1</span><span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
 <span class="token punctuation">}</span>
 <span class="token punctuation">}</span>
<span class="token punctuation">}</span>
</code></pre></div><p>reducer</p> <div class="language-java extra-class"><pre class="language-java"><code><span class="token keyword">public</span> <span class="token keyword">class</span> <span class="token class-name">WordCountReducer</span> <span class="token keyword">extends</span>
<span class="token class-name">Reducer</span><span class="token generics"><span class="token punctuation">&lt;</span><span class="token class-name">Text</span><span class="token punctuation">,</span><span class="token class-name">IntWritable</span><span class="token punctuation">,</span><span class="token class-name">Text</span><span class="token punctuation">,</span><span class="token class-name">IntWritable</span><span class="token punctuation">&gt;</span></span> <span class="token punctuation">{</span>
 <span class="token comment">//key-&gt;单词 values-&gt;次数 1 1 1 1 1</span>
 <span class="token annotation punctuation">@Override</span>
 <span class="token keyword">protected</span> <span class="token keyword">void</span> <span class="token function">reduce</span><span class="token punctuation">(</span><span class="token class-name">Text</span> key<span class="token punctuation">,</span> <span class="token class-name">Iterable</span><span class="token generics"><span class="token punctuation">&lt;</span><span class="token class-name">IntWritable</span><span class="token punctuation">&gt;</span></span> values<span class="token punctuation">,</span> <span class="token class-name">Context</span>
context<span class="token punctuation">)</span> <span class="token keyword">throws</span> <span class="token class-name">IOException</span><span class="token punctuation">,</span> <span class="token class-name">InterruptedException</span> <span class="token punctuation">{</span>
 <span class="token comment">//1.记录出现的次数</span>
 <span class="token keyword">int</span> sum <span class="token operator">=</span> <span class="token number">0</span><span class="token punctuation">;</span>
 <span class="token keyword">for</span><span class="token punctuation">(</span><span class="token class-name">IntWritable</span> v<span class="token operator">:</span>values<span class="token punctuation">)</span><span class="token punctuation">{</span>
 sum <span class="token operator">+=</span> v<span class="token punctuation">.</span><span class="token function">get</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
 <span class="token punctuation">}</span>
 <span class="token comment">//2.累加求和输出</span>
 context<span class="token punctuation">.</span><span class="token function">write</span><span class="token punctuation">(</span>key<span class="token punctuation">,</span><span class="token keyword">new</span> <span class="token class-name">IntWritable</span><span class="token punctuation">(</span>sum<span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
 <span class="token punctuation">}</span>
<span class="token punctuation">}</span>

</code></pre></div><p>main</p> <div class="language-java extra-class"><pre class="language-java"><code><span class="token keyword">public</span> <span class="token keyword">class</span> <span class="token class-name">WordCountDriver</span> <span class="token punctuation">{</span>
 <span class="token keyword">public</span> <span class="token keyword">static</span> <span class="token keyword">void</span> <span class="token function">main</span><span class="token punctuation">(</span><span class="token class-name">String</span><span class="token punctuation">[</span><span class="token punctuation">]</span> args<span class="token punctuation">)</span> <span class="token keyword">throws</span> <span class="token class-name">IOException</span><span class="token punctuation">,</span> <span class="token class-name">ClassNotFoundException</span><span class="token punctuation">,</span> <span class="token class-name">InterruptedException</span>
<span class="token punctuation">{</span>
 <span class="token class-name">Configuration</span> conf <span class="token operator">=</span> <span class="token keyword">new</span> <span class="token class-name">Configuration</span><span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
 <span class="token class-name">Job</span> job <span class="token operator">=</span> <span class="token class-name">Job</span><span class="token punctuation">.</span><span class="token function">getInstance</span><span class="token punctuation">(</span>conf<span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token comment">//1.创建job任务</span>
 job<span class="token punctuation">.</span><span class="token function">setJarByClass</span><span class="token punctuation">(</span><span class="token class-name">WordCountDriver</span><span class="token punctuation">.</span><span class="token keyword">class</span><span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token comment">//2.指定jar包位置</span>
 job<span class="token punctuation">.</span><span class="token function">setMapperClass</span><span class="token punctuation">(</span><span class="token class-name">WordCountMapper</span><span class="token punctuation">.</span><span class="token keyword">class</span><span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token comment">//3.关联使用的Mapper类</span>
 job<span class="token punctuation">.</span><span class="token function">setReducerClass</span><span class="token punctuation">(</span><span class="token class-name">WordCountReducer</span><span class="token punctuation">.</span><span class="token keyword">class</span><span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token comment">//4.关联使用的Reducer类</span>
 job<span class="token punctuation">.</span><span class="token function">setMapOutputKeyClass</span><span class="token punctuation">(</span><span class="token class-name">Text</span><span class="token punctuation">.</span><span class="token keyword">class</span><span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token comment">//5.设置mapper阶段输出的数据类型</span>
 job<span class="token punctuation">.</span><span class="token function">setMapOutputValueClass</span><span class="token punctuation">(</span><span class="token class-name">IntWritable</span><span class="token punctuation">.</span><span class="token keyword">class</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
 job<span class="token punctuation">.</span><span class="token function">setOutputKeyClass</span><span class="token punctuation">(</span><span class="token class-name">Text</span><span class="token punctuation">.</span><span class="token keyword">class</span><span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token comment">//6.设置reducer阶段输出的数据类型</span>
 job<span class="token punctuation">.</span><span class="token function">setOutputValueClass</span><span class="token punctuation">(</span><span class="token class-name">IntWritable</span><span class="token punctuation">.</span><span class="token keyword">class</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
 <span class="token class-name">FileInputFormat</span><span class="token punctuation">.</span><span class="token function">setInputPaths</span><span class="token punctuation">(</span>job<span class="token punctuation">,</span><span class="token keyword">new</span> <span class="token class-name">Path</span><span class="token punctuation">(</span>args<span class="token punctuation">[</span><span class="token number">0</span><span class="token punctuation">]</span><span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token comment">//7.设置数据输入的路径</span>
 <span class="token class-name">FileOutputFormat</span><span class="token punctuation">.</span><span class="token function">setOutputPath</span><span class="token punctuation">(</span>job<span class="token punctuation">,</span><span class="token keyword">new</span> <span class="token class-name">Path</span><span class="token punctuation">(</span>args<span class="token punctuation">[</span><span class="token number">1</span><span class="token punctuation">]</span><span class="token punctuation">)</span><span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token comment">//8.设置数据输出的路径</span>
 <span class="token keyword">boolean</span> rs <span class="token operator">=</span> job<span class="token punctuation">.</span><span class="token function">waitForCompletion</span><span class="token punctuation">(</span><span class="token boolean">true</span><span class="token punctuation">)</span><span class="token punctuation">;</span> <span class="token comment">//9.提交任务</span>
 <span class="token class-name">System</span><span class="token punctuation">.</span><span class="token function">exit</span><span class="token punctuation">(</span>rs<span class="token operator">?</span><span class="token number">0</span><span class="token operator">:</span><span class="token number">1</span><span class="token punctuation">)</span><span class="token punctuation">;</span>
 <span class="token punctuation">}</span>
<span class="token punctuation">}</span>
</code></pre></div><h3 id="_7-了解zookeeper架构和工作原理"><a href="#_7-了解zookeeper架构和工作原理" class="header-anchor">#</a> （7）了解zookeeper架构和工作原理</h3> <p><img src="/assets/img/image-20200626132601984.84dd89f0.png" alt="image-20200626132601984"></p> <p><img src="/assets/img/image-20200626132750824.4f55aa92.png" alt="image-20200626132750824"></p> <p>Zookeeper的核心是原子广播，这个机制保证了各个server之间的同步。</p> <p>实现这个机制的协议叫做Zab协议。</p> <p>Zab协议有两种模式，它们分别是恢复模式和广播模式。</p> <p>当服务启动或者在领导者崩溃后，Zab就进入了恢复模式，当领导者被选举出来，且大多数server的完成 了和leader的状态同步以后，恢复模式就结束了。</p> <p>状态同步保证了leader和server具有相同的系统状态。</p> <h2 id="大数据采集与预处理"><a href="#大数据采集与预处理" class="header-anchor">#</a> 大数据采集与预处理</h2> <h3 id="_1-理解数据及其分类"><a href="#_1-理解数据及其分类" class="header-anchor">#</a> （1）理解数据及其分类；</h3> <p>数据是对研究现象的属性和特征的具体描述</p> <p>分类：</p> <ol><li>按结构化程度分：结构化，半结构化，非结构化</li> <li>按数据加工程度分：裸数据，专家数据，信息，价值</li> <li>价值角度分：流动数据，静态数据</li></ol> <h3 id="_2-了解数据采集方法和工具"><a href="#_2-了解数据采集方法和工具" class="header-anchor">#</a> （2）了解数据采集方法和工具；</h3> <p>采集方法</p> <ol><li>数据抓取</li> <li>数据导入</li> <li>物联网传感设备自动信息采集</li></ol> <p>日志系统采集工具：chukwa、flume、scribe、kafka</p> <h3 id="_3-掌握数据清洗的任务和作用"><a href="#_3-掌握数据清洗的任务和作用" class="header-anchor">#</a> （3）掌握数据清洗的任务和作用；</h3> <p>任务是发现并纠正数据文件中可识别的错误，包括对数据一致性的检查，无效值和缺失值得处理。</p> <p>数据清洗的作用是将脏数据转化为满足数据质量要求的数据</p> <h3 id="_4-掌握数据变换的种类和过程"><a href="#_4-掌握数据变换的种类和过程" class="header-anchor">#</a> （4）掌握数据变换的种类和过程；</h3> <p>数据变换：将多维数据压缩成较少维的数据，消除时间、空间、属性及精度等特征表现的差异</p> <p><strong>中心化变换</strong></p> <p><span class="katex"><span class="katex-mathml"><math><semantics><mrow><msubsup><mi>x</mi><mrow><mi>i</mi><mi>j</mi></mrow><mo>∗</mo></msubsup><mo>=</mo><msub><mi>x</mi><mrow><mi>i</mi><mi>j</mi></mrow></msub><mo>−</mo><msub><mover accent="true"><mi>x</mi><mo>¯</mo></mover><mi>j</mi></msub><mo separator="true">,</mo><mi>i</mi><mo>=</mo><mn>1</mn><mo separator="true">,</mo><mn>2</mn><mo separator="true">,</mo><mo>⋯</mo><mo separator="true">,</mo><mi>n</mi><mo separator="true">,</mo><mi>j</mi><mo>=</mo><mn>1</mn><mo separator="true">,</mo><mn>2</mn><mo separator="true">,</mo><mo>⋯</mo><mo separator="true">,</mo><mi>m</mi></mrow><annotation encoding="application/x-tex">x_{ij}^* = x_{ij} - \bar x_j, i=1,2,\cdots, n, j=1,2,\cdots, m</annotation></semantics></math></span><span aria-hidden="true" class="katex-html"><span class="strut" style="height:0.688696em;"></span><span class="strut bottom" style="height:1.071804em;vertical-align:-0.383108em;"></span><span class="base textstyle uncramped"><span class="mord"><span class="mord mathit">x</span><span class="vlist"><span style="top:0.24700000000000003em;margin-left:0em;margin-right:0.05em;"><span class="fontsize-ensurer reset-size5 size5"><span style="font-size:0em;">​</span></span><span class="reset-textstyle scriptstyle cramped"><span class="mord scriptstyle cramped"><span class="mord mathit">i</span><span class="mord mathit" style="margin-right:0.05724em;">j</span></span></span></span><span style="top:-0.363em;margin-right:0.05em;"><span class="fontsize-ensurer reset-size5 size5"><span style="font-size:0em;">​</span></span><span class="reset-textstyle scriptstyle uncramped"><span class="mord">∗</span></span></span><span class="baseline-fix"><span class="fontsize-ensurer reset-size5 size5"><span style="font-size:0em;">​</span></span>​</span></span></span><span class="mrel">=</span><span class="mord"><span class="mord mathit">x</span><span class="vlist"><span style="top:0.15em;margin-right:0.05em;margin-left:0em;"><span class="fontsize-ensurer reset-size5 size5"><span style="font-size:0em;">​</span></span><span class="reset-textstyle scriptstyle cramped"><span class="mord scriptstyle cramped"><span class="mord mathit">i</span><span class="mord mathit" style="margin-right:0.05724em;">j</span></span></span></span><span class="baseline-fix"><span class="fontsize-ensurer reset-size5 size5"><span style="font-size:0em;">​</span></span>​</span></span></span><span class="mbin">−</span><span class="mord"><span class="mord accent"><span class="vlist"><span style="top:0em;"><span class="fontsize-ensurer reset-size5 size5"><span style="font-size:0em;">​</span></span><span class="mord mathit">x</span></span><span style="top:0em;margin-left:0.05556em;"><span class="fontsize-ensurer reset-size5 size5"><span style="font-size:0em;">​</span></span><span class="accent-body"><span>¯</span></span></span><span class="baseline-fix"><span class="fontsize-ensurer reset-size5 size5"><span style="font-size:0em;">​</span></span>​</span></span></span><span class="vlist"><span style="top:0.15em;margin-right:0.05em;margin-left:0em;"><span class="fontsize-ensurer reset-size5 size5"><span style="font-size:0em;">​</span></span><span class="reset-textstyle scriptstyle cramped"><span class="mord mathit" style="margin-right:0.05724em;">j</span></span></span><span class="baseline-fix"><span class="fontsize-ensurer reset-size5 size5"><span style="font-size:0em;">​</span></span>​</span></span></span><span class="mpunct">,</span><span class="mord mathit">i</span><span class="mrel">=</span><span class="mord mathrm">1</span><span class="mpunct">,</span><span class="mord mathrm">2</span><span class="mpunct">,</span><span class="minner">⋯</span><span class="mpunct">,</span><span class="mord mathit">n</span><span class="mpunct">,</span><span class="mord mathit" style="margin-right:0.05724em;">j</span><span class="mrel">=</span><span class="mord mathrm">1</span><span class="mpunct">,</span><span class="mord mathrm">2</span><span class="mpunct">,</span><span class="minner">⋯</span><span class="mpunct">,</span><span class="mord mathit">m</span></span></span></span></p> <p><span class="katex"><span class="katex-mathml"><math><semantics><mrow><msup><mi>S</mi><mo>∗</mo></msup><mo>=</mo><mi>S</mi><mo>=</mo><mo>(</mo><msub><mi>s</mi><mrow><mi>i</mi><mi>j</mi></mrow></msub><msub><mo>)</mo><mrow><mi>m</mi><mo>×</mo><mi>m</mi></mrow></msub></mrow><annotation encoding="application/x-tex">S^* = S = (s_{ij})_{m\times m}</annotation></semantics></math></span><span aria-hidden="true" class="katex-html"><span class="strut" style="height:0.75em;"></span><span class="strut bottom" style="height:1.036108em;vertical-align:-0.286108em;"></span><span class="base textstyle uncramped"><span class="mord"><span class="mord mathit" style="margin-right:0.05764em;">S</span><span class="vlist"><span style="top:-0.363em;margin-right:0.05em;"><span class="fontsize-ensurer reset-size5 size5"><span style="font-size:0em;">​</span></span><span class="reset-textstyle scriptstyle uncramped"><span class="mord">∗</span></span></span><span class="baseline-fix"><span class="fontsize-ensurer reset-size5 size5"><span style="font-size:0em;">​</span></span>​</span></span></span><span class="mrel">=</span><span class="mord mathit" style="margin-right:0.05764em;">S</span><span class="mrel">=</span><span class="mopen">(</span><span class="mord"><span class="mord mathit">s</span><span class="vlist"><span style="top:0.15em;margin-right:0.05em;margin-left:0em;"><span class="fontsize-ensurer reset-size5 size5"><span style="font-size:0em;">​</span></span><span class="reset-textstyle scriptstyle cramped"><span class="mord scriptstyle cramped"><span class="mord mathit">i</span><span class="mord mathit" style="margin-right:0.05724em;">j</span></span></span></span><span class="baseline-fix"><span class="fontsize-ensurer reset-size5 size5"><span style="font-size:0em;">​</span></span>​</span></span></span><span class="mclose"><span class="mclose">)</span><span class="vlist"><span style="top:0.15em;margin-right:0.05em;margin-left:0em;"><span class="fontsize-ensurer reset-size5 size5"><span style="font-size:0em;">​</span></span><span class="reset-textstyle scriptstyle cramped"><span class="mord scriptstyle cramped"><span class="mord mathit">m</span><span class="mbin">×</span><span class="mord mathit">m</span></span></span></span><span class="baseline-fix"><span class="fontsize-ensurer reset-size5 size5"><span style="font-size:0em;">​</span></span>​</span></span></span></span></span></span></p> <p>其中 <span class="katex"><span class="katex-mathml"><math><semantics><mrow><msub><mi>s</mi><mrow><mi>i</mi><mi>j</mi></mrow></msub><mo>=</mo><mfrac><mrow><mn>1</mn></mrow><mrow><mi>n</mi><mo>−</mo><mn>1</mn></mrow></mfrac><msubsup><mo>∑</mo><mrow><mi>t</mi><mo>=</mo><mn>1</mn></mrow><mi>n</mi></msubsup><mo>(</mo><msubsup><mi>x</mi><mrow><mi>t</mi><mi>i</mi></mrow><mo>∗</mo></msubsup><msubsup><mi>x</mi><mrow><mi>t</mi><mi>j</mi></mrow><mo>∗</mo></msubsup><mo>)</mo></mrow><annotation encoding="application/x-tex">s_{ij} = \frac{1}{n-1}\sum_{t=1}^n (x^*_{ti}x^*_{tj})</annotation></semantics></math></span><span aria-hidden="true" class="katex-html"><span class="strut" style="height:0.845108em;"></span><span class="strut bottom" style="height:1.2484389999999999em;vertical-align:-0.403331em;"></span><span class="base textstyle uncramped"><span class="mord"><span class="mord mathit">s</span><span class="vlist"><span style="top:0.15em;margin-right:0.05em;margin-left:0em;"><span class="fontsize-ensurer reset-size5 size5"><span style="font-size:0em;">​</span></span><span class="reset-textstyle scriptstyle cramped"><span class="mord scriptstyle cramped"><span class="mord mathit">i</span><span class="mord mathit" style="margin-right:0.05724em;">j</span></span></span></span><span class="baseline-fix"><span class="fontsize-ensurer reset-size5 size5"><span style="font-size:0em;">​</span></span>​</span></span></span><span class="mrel">=</span><span class="mord reset-textstyle textstyle uncramped"><span class="sizing reset-size5 size5 reset-textstyle textstyle uncramped nulldelimiter"></span><span class="mfrac"><span class="vlist"><span style="top:0.345em;"><span class="fontsize-ensurer reset-size5 size5"><span style="font-size:0em;">​</span></span><span class="reset-textstyle scriptstyle cramped"><span class="mord scriptstyle cramped"><span class="mord mathit">n</span><span class="mbin">−</span><span class="mord mathrm">1</span></span></span></span><span style="top:-0.22999999999999998em;"><span class="fontsize-ensurer reset-size5 size5"><span style="font-size:0em;">​</span></span><span class="reset-textstyle textstyle uncramped frac-line"></span></span><span style="top:-0.394em;"><span class="fontsize-ensurer reset-size5 size5"><span style="font-size:0em;">​</span></span><span class="reset-textstyle scriptstyle uncramped"><span class="mord scriptstyle uncramped"><span class="mord mathrm">1</span></span></span></span><span class="baseline-fix"><span class="fontsize-ensurer reset-size5 size5"><span style="font-size:0em;">​</span></span>​</span></span></span><span class="sizing reset-size5 size5 reset-textstyle textstyle uncramped nulldelimiter"></span></span><span class="mop"><span class="op-symbol small-op mop" style="top:-0.0000050000000000050004em;">∑</span><span class="vlist"><span style="top:0.30001em;margin-left:0em;margin-right:0.05em;"><span class="fontsize-ensurer reset-size5 size5"><span style="font-size:0em;">​</span></span><span class="reset-textstyle scriptstyle cramped"><span class="mord scriptstyle cramped"><span class="mord mathit">t</span><span class="mrel">=</span><span class="mord mathrm">1</span></span></span></span><span style="top:-0.364em;margin-right:0.05em;"><span class="fontsize-ensurer reset-size5 size5"><span style="font-size:0em;">​</span></span><span class="reset-textstyle scriptstyle uncramped"><span class="mord mathit">n</span></span></span><span class="baseline-fix"><span class="fontsize-ensurer reset-size5 size5"><span style="font-size:0em;">​</span></span>​</span></span></span><span class="mopen">(</span><span class="mord"><span class="mord mathit">x</span><span class="vlist"><span style="top:0.247em;margin-left:0em;margin-right:0.05em;"><span class="fontsize-ensurer reset-size5 size5"><span style="font-size:0em;">​</span></span><span class="reset-textstyle scriptstyle cramped"><span class="mord scriptstyle cramped"><span class="mord mathit">t</span><span class="mord mathit">i</span></span></span></span><span style="top:-0.363em;margin-right:0.05em;"><span class="fontsize-ensurer reset-size5 size5"><span style="font-size:0em;">​</span></span><span class="reset-textstyle scriptstyle uncramped"><span class="mord">∗</span></span></span><span class="baseline-fix"><span class="fontsize-ensurer reset-size5 size5"><span style="font-size:0em;">​</span></span>​</span></span></span><span class="mord"><span class="mord mathit">x</span><span class="vlist"><span style="top:0.24700000000000003em;margin-left:0em;margin-right:0.05em;"><span class="fontsize-ensurer reset-size5 size5"><span style="font-size:0em;">​</span></span><span class="reset-textstyle scriptstyle cramped"><span class="mord scriptstyle cramped"><span class="mord mathit">t</span><span class="mord mathit" style="margin-right:0.05724em;">j</span></span></span></span><span style="top:-0.363em;margin-right:0.05em;"><span class="fontsize-ensurer reset-size5 size5"><span style="font-size:0em;">​</span></span><span class="reset-textstyle scriptstyle uncramped"><span class="mord">∗</span></span></span><span class="baseline-fix"><span class="fontsize-ensurer reset-size5 size5"><span style="font-size:0em;">​</span></span>​</span></span></span><span class="mclose">)</span></span></span></span></p> <p><strong>作用</strong>：变换之后均值为0，协方差阵不变，可以用来方便地计算样本协方差阵。</p> <p><strong>标准化变换</strong></p> <p><span class="katex"><span class="katex-mathml"><math><semantics><mrow><msubsup><mi>x</mi><mrow><mi>i</mi><mi>j</mi></mrow><mo>∗</mo></msubsup><mo>=</mo><mfrac><mrow><msub><mi>x</mi><mrow><mi>i</mi><mi>j</mi></mrow></msub><mo>−</mo><msub><mover accent="true"><mi>x</mi><mo>¯</mo></mover><mi>j</mi></msub></mrow><mrow><msub><mi>s</mi><mrow><mi>j</mi></mrow></msub></mrow></mfrac><mo separator="true">,</mo><mi>i</mi><mo>=</mo><mn>1</mn><mo separator="true">,</mo><mn>2</mn><mo separator="true">,</mo><mo>⋯</mo><mo separator="true">,</mo><mi>n</mi><mo separator="true">,</mo><mi>j</mi><mo>=</mo><mn>1</mn><mo separator="true">,</mo><mn>2</mn><mo separator="true">,</mo><mo>⋯</mo><mo separator="true">,</mo><mi>m</mi></mrow><annotation encoding="application/x-tex">x_{ij}^* = \frac{x_{ij} - \bar x_j}{s_{j}}, i=1,2,\cdots, n, j = 1,2,\cdots, m</annotation></semantics></math></span><span aria-hidden="true" class="katex-html"><span class="strut" style="height:0.9205510000000001em;"></span><span class="strut bottom" style="height:1.4677710000000002em;vertical-align:-0.54722em;"></span><span class="base textstyle uncramped"><span class="mord"><span class="mord mathit">x</span><span class="vlist"><span style="top:0.24700000000000003em;margin-left:0em;margin-right:0.05em;"><span class="fontsize-ensurer reset-size5 size5"><span style="font-size:0em;">​</span></span><span class="reset-textstyle scriptstyle cramped"><span class="mord scriptstyle cramped"><span class="mord mathit">i</span><span class="mord mathit" style="margin-right:0.05724em;">j</span></span></span></span><span style="top:-0.363em;margin-right:0.05em;"><span class="fontsize-ensurer reset-size5 size5"><span style="font-size:0em;">​</span></span><span class="reset-textstyle scriptstyle uncramped"><span class="mord">∗</span></span></span><span class="baseline-fix"><span class="fontsize-ensurer reset-size5 size5"><span style="font-size:0em;">​</span></span>​</span></span></span><span class="mrel">=</span><span class="mord reset-textstyle textstyle uncramped"><span class="sizing reset-size5 size5 reset-textstyle textstyle uncramped nulldelimiter"></span><span class="mfrac"><span class="vlist"><span style="top:0.34500000000000003em;"><span class="fontsize-ensurer reset-size5 size5"><span style="font-size:0em;">​</span></span><span class="reset-textstyle scriptstyle cramped"><span class="mord scriptstyle cramped"><span class="mord"><span class="mord mathit">s</span><span class="vlist"><span style="top:0.15000000000000002em;margin-right:0.07142857142857144em;margin-left:0em;"><span class="fontsize-ensurer reset-size5 size5"><span style="font-size:0em;">​</span></span><span class="reset-scriptstyle scriptscriptstyle cramped"><span class="mord scriptscriptstyle cramped"><span class="mord mathit" style="margin-right:0.05724em;">j</span></span></span></span><span class="baseline-fix"><span class="fontsize-ensurer reset-size5 size5"><span style="font-size:0em;">​</span></span>​</span></span></span></span></span></span><span style="top:-0.22999999999999998em;"><span class="fontsize-ensurer reset-size5 size5"><span style="font-size:0em;">​</span></span><span class="reset-textstyle textstyle uncramped frac-line"></span></span><span style="top:-0.51222em;"><span class="fontsize-ensurer reset-size5 size5"><span style="font-size:0em;">​</span></span><span class="reset-textstyle scriptstyle uncramped"><span class="mord scriptstyle uncramped"><span class="mord"><span class="mord mathit">x</span><span class="vlist"><span style="top:0.15000000000000002em;margin-right:0.07142857142857144em;margin-left:0em;"><span class="fontsize-ensurer reset-size5 size5"><span style="font-size:0em;">​</span></span><span class="reset-scriptstyle scriptscriptstyle cramped"><span class="mord scriptscriptstyle cramped"><span class="mord mathit">i</span><span class="mord mathit" style="margin-right:0.05724em;">j</span></span></span></span><span class="baseline-fix"><span class="fontsize-ensurer reset-size5 size5"><span style="font-size:0em;">​</span></span>​</span></span></span><span class="mbin">−</span><span class="mord"><span class="mord accent"><span class="vlist"><span style="top:0em;"><span class="fontsize-ensurer reset-size5 size5"><span style="font-size:0em;">​</span></span><span class="mord mathit">x</span></span><span style="top:0em;margin-left:0.05556em;"><span class="fontsize-ensurer reset-size5 size5"><span style="font-size:0em;">​</span></span><span class="accent-body"><span>¯</span></span></span><span class="baseline-fix"><span class="fontsize-ensurer reset-size5 size5"><span style="font-size:0em;">​</span></span>​</span></span></span><span class="vlist"><span style="top:0.15000000000000002em;margin-right:0.07142857142857144em;margin-left:0em;"><span class="fontsize-ensurer reset-size5 size5"><span style="font-size:0em;">​</span></span><span class="reset-scriptstyle scriptscriptstyle cramped"><span class="mord mathit" style="margin-right:0.05724em;">j</span></span></span><span class="baseline-fix"><span class="fontsize-ensurer reset-size5 size5"><span style="font-size:0em;">​</span></span>​</span></span></span></span></span></span><span class="baseline-fix"><span class="fontsize-ensurer reset-size5 size5"><span style="font-size:0em;">​</span></span>​</span></span></span><span class="sizing reset-size5 size5 reset-textstyle textstyle uncramped nulldelimiter"></span></span><span class="mpunct">,</span><span class="mord mathit">i</span><span class="mrel">=</span><span class="mord mathrm">1</span><span class="mpunct">,</span><span class="mord mathrm">2</span><span class="mpunct">,</span><span class="minner">⋯</span><span class="mpunct">,</span><span class="mord mathit">n</span><span class="mpunct">,</span><span class="mord mathit" style="margin-right:0.05724em;">j</span><span class="mrel">=</span><span class="mord mathrm">1</span><span class="mpunct">,</span><span class="mord mathrm">2</span><span class="mpunct">,</span><span class="minner">⋯</span><span class="mpunct">,</span><span class="mord mathit">m</span></span></span></span></p> <p><strong>作用</strong>：变换之后每个变量均值为0，标准差为1，变换后的数据与变量的量纲无关。</p> <p><strong>极差标准化变换</strong></p> <p><span class="katex"><span class="katex-mathml"><math><semantics><mrow><msubsup><mi>x</mi><mrow><mi>i</mi><mi>j</mi></mrow><mo>∗</mo></msubsup><mo>=</mo><mfrac><mrow><msub><mi>x</mi><mrow><mi>i</mi><mi>j</mi></mrow></msub><mo>−</mo><msub><mover accent="true"><mi>x</mi><mo>¯</mo></mover><mi>j</mi></msub></mrow><mrow><msub><mi>R</mi><mrow><mi>j</mi></mrow></msub></mrow></mfrac><mo separator="true">,</mo><mi>i</mi><mo>=</mo><mn>1</mn><mo separator="true">,</mo><mn>2</mn><mo separator="true">,</mo><mo>⋯</mo><mo separator="true">,</mo><mi>n</mi><mo separator="true">,</mo><mi>j</mi><mo>=</mo><mn>1</mn><mo separator="true">,</mo><mn>2</mn><mo separator="true">,</mo><mo>⋯</mo><mo separator="true">,</mo><mi>m</mi></mrow><annotation encoding="application/x-tex">x_{ij}^* = \frac{x_{ij} - \bar x_j}{R_{j}}, i=1,2,\cdots, n, j = 1,2,\cdots, m</annotation></semantics></math></span><span aria-hidden="true" class="katex-html"><span class="strut" style="height:0.9205510000000001em;"></span><span class="strut bottom" style="height:1.4677710000000002em;vertical-align:-0.54722em;"></span><span class="base textstyle uncramped"><span class="mord"><span class="mord mathit">x</span><span class="vlist"><span style="top:0.24700000000000003em;margin-left:0em;margin-right:0.05em;"><span class="fontsize-ensurer reset-size5 size5"><span style="font-size:0em;">​</span></span><span class="reset-textstyle scriptstyle cramped"><span class="mord scriptstyle cramped"><span class="mord mathit">i</span><span class="mord mathit" style="margin-right:0.05724em;">j</span></span></span></span><span style="top:-0.363em;margin-right:0.05em;"><span class="fontsize-ensurer reset-size5 size5"><span style="font-size:0em;">​</span></span><span class="reset-textstyle scriptstyle uncramped"><span class="mord">∗</span></span></span><span class="baseline-fix"><span class="fontsize-ensurer reset-size5 size5"><span style="font-size:0em;">​</span></span>​</span></span></span><span class="mrel">=</span><span class="mord reset-textstyle textstyle uncramped"><span class="sizing reset-size5 size5 reset-textstyle textstyle uncramped nulldelimiter"></span><span class="mfrac"><span class="vlist"><span style="top:0.34500000000000003em;"><span class="fontsize-ensurer reset-size5 size5"><span style="font-size:0em;">​</span></span><span class="reset-textstyle scriptstyle cramped"><span class="mord scriptstyle cramped"><span class="mord"><span class="mord mathit" style="margin-right:0.00773em;">R</span><span class="vlist"><span style="top:0.15000000000000002em;margin-right:0.07142857142857144em;margin-left:-0.00773em;"><span class="fontsize-ensurer reset-size5 size5"><span style="font-size:0em;">​</span></span><span class="reset-scriptstyle scriptscriptstyle cramped"><span class="mord scriptscriptstyle cramped"><span class="mord mathit" style="margin-right:0.05724em;">j</span></span></span></span><span class="baseline-fix"><span class="fontsize-ensurer reset-size5 size5"><span style="font-size:0em;">​</span></span>​</span></span></span></span></span></span><span style="top:-0.22999999999999998em;"><span class="fontsize-ensurer reset-size5 size5"><span style="font-size:0em;">​</span></span><span class="reset-textstyle textstyle uncramped frac-line"></span></span><span style="top:-0.51222em;"><span class="fontsize-ensurer reset-size5 size5"><span style="font-size:0em;">​</span></span><span class="reset-textstyle scriptstyle uncramped"><span class="mord scriptstyle uncramped"><span class="mord"><span class="mord mathit">x</span><span class="vlist"><span style="top:0.15000000000000002em;margin-right:0.07142857142857144em;margin-left:0em;"><span class="fontsize-ensurer reset-size5 size5"><span style="font-size:0em;">​</span></span><span class="reset-scriptstyle scriptscriptstyle cramped"><span class="mord scriptscriptstyle cramped"><span class="mord mathit">i</span><span class="mord mathit" style="margin-right:0.05724em;">j</span></span></span></span><span class="baseline-fix"><span class="fontsize-ensurer reset-size5 size5"><span style="font-size:0em;">​</span></span>​</span></span></span><span class="mbin">−</span><span class="mord"><span class="mord accent"><span class="vlist"><span style="top:0em;"><span class="fontsize-ensurer reset-size5 size5"><span style="font-size:0em;">​</span></span><span class="mord mathit">x</span></span><span style="top:0em;margin-left:0.05556em;"><span class="fontsize-ensurer reset-size5 size5"><span style="font-size:0em;">​</span></span><span class="accent-body"><span>¯</span></span></span><span class="baseline-fix"><span class="fontsize-ensurer reset-size5 size5"><span style="font-size:0em;">​</span></span>​</span></span></span><span class="vlist"><span style="top:0.15000000000000002em;margin-right:0.07142857142857144em;margin-left:0em;"><span class="fontsize-ensurer reset-size5 size5"><span style="font-size:0em;">​</span></span><span class="reset-scriptstyle scriptscriptstyle cramped"><span class="mord mathit" style="margin-right:0.05724em;">j</span></span></span><span class="baseline-fix"><span class="fontsize-ensurer reset-size5 size5"><span style="font-size:0em;">​</span></span>​</span></span></span></span></span></span><span class="baseline-fix"><span class="fontsize-ensurer reset-size5 size5"><span style="font-size:0em;">​</span></span>​</span></span></span><span class="sizing reset-size5 size5 reset-textstyle textstyle uncramped nulldelimiter"></span></span><span class="mpunct">,</span><span class="mord mathit">i</span><span class="mrel">=</span><span class="mord mathrm">1</span><span class="mpunct">,</span><span class="mord mathrm">2</span><span class="mpunct">,</span><span class="minner">⋯</span><span class="mpunct">,</span><span class="mord mathit">n</span><span class="mpunct">,</span><span class="mord mathit" style="margin-right:0.05724em;">j</span><span class="mrel">=</span><span class="mord mathrm">1</span><span class="mpunct">,</span><span class="mord mathrm">2</span><span class="mpunct">,</span><span class="minner">⋯</span><span class="mpunct">,</span><span class="mord mathit">m</span></span></span></span></p> <p>其中 $R_j= \max_{1\le t \le n}x_{tj} - \min_{1\le t \le n}x_{tj} $</p> <p><strong>作用</strong>：变换后每个变量样本均值为0，极差为1，变换后数据绝对值数据在(-1,1) 中，能减少分析计算中的误差，无量纲。</p> <p><strong>极差正规化变换</strong></p> <p><span class="katex"><span class="katex-mathml"><math><semantics><mrow><msubsup><mi>x</mi><mrow><mi>i</mi><mi>j</mi></mrow><mo>∗</mo></msubsup><mo>=</mo><mfrac><mrow><msub><mi>x</mi><mrow><mi>i</mi><mi>j</mi></mrow></msub><mo>−</mo><msub><mi>min</mi><mrow><mn>1</mn><mo>≤</mo><mi>t</mi><mo>≤</mo><mi>n</mi></mrow></msub><msub><mi>x</mi><mrow><mi>t</mi><mi>j</mi></mrow></msub></mrow><mrow><msub><mi>R</mi><mi>j</mi></msub></mrow></mfrac><mo separator="true">,</mo><mi>i</mi><mo>=</mo><mn>1</mn><mo separator="true">,</mo><mn>2</mn><mo separator="true">,</mo><mo>⋯</mo><mo separator="true">,</mo><mi>n</mi><mo separator="true">,</mo><mi>j</mi><mo>=</mo><mn>1</mn><mo separator="true">,</mo><mn>2</mn><mo separator="true">,</mo><mo>⋯</mo><mo separator="true">,</mo><mi>m</mi></mrow><annotation encoding="application/x-tex">x_{ij}^* = \frac{x_{ij} - \min_{1 \le t \le n} x_{tj} }{R_j}, i=1,2,\cdots, n, j = 1,2,\cdots, m</annotation></semantics></math></span><span aria-hidden="true" class="katex-html"><span class="strut" style="height:0.979722em;"></span><span class="strut bottom" style="height:1.526942em;vertical-align:-0.54722em;"></span><span class="base textstyle uncramped"><span class="mord"><span class="mord mathit">x</span><span class="vlist"><span style="top:0.24700000000000003em;margin-left:0em;margin-right:0.05em;"><span class="fontsize-ensurer reset-size5 size5"><span style="font-size:0em;">​</span></span><span class="reset-textstyle scriptstyle cramped"><span class="mord scriptstyle cramped"><span class="mord mathit">i</span><span class="mord mathit" style="margin-right:0.05724em;">j</span></span></span></span><span style="top:-0.363em;margin-right:0.05em;"><span class="fontsize-ensurer reset-size5 size5"><span style="font-size:0em;">​</span></span><span class="reset-textstyle scriptstyle uncramped"><span class="mord">∗</span></span></span><span class="baseline-fix"><span class="fontsize-ensurer reset-size5 size5"><span style="font-size:0em;">​</span></span>​</span></span></span><span class="mrel">=</span><span class="mord reset-textstyle textstyle uncramped"><span class="sizing reset-size5 size5 reset-textstyle textstyle uncramped nulldelimiter"></span><span class="mfrac"><span class="vlist"><span style="top:0.34500000000000003em;"><span class="fontsize-ensurer reset-size5 size5"><span style="font-size:0em;">​</span></span><span class="reset-textstyle scriptstyle cramped"><span class="mord scriptstyle cramped"><span class="mord"><span class="mord mathit" style="margin-right:0.00773em;">R</span><span class="vlist"><span style="top:0.15000000000000002em;margin-right:0.07142857142857144em;margin-left:-0.00773em;"><span class="fontsize-ensurer reset-size5 size5"><span style="font-size:0em;">​</span></span><span class="reset-scriptstyle scriptscriptstyle cramped"><span class="mord mathit" style="margin-right:0.05724em;">j</span></span></span><span class="baseline-fix"><span class="fontsize-ensurer reset-size5 size5"><span style="font-size:0em;">​</span></span>​</span></span></span></span></span></span><span style="top:-0.22999999999999998em;"><span class="fontsize-ensurer reset-size5 size5"><span style="font-size:0em;">​</span></span><span class="reset-textstyle textstyle uncramped frac-line"></span></span><span style="top:-0.51222em;"><span class="fontsize-ensurer reset-size5 size5"><span style="font-size:0em;">​</span></span><span class="reset-textstyle scriptstyle uncramped"><span class="mord scriptstyle uncramped"><span class="mord"><span class="mord mathit">x</span><span class="vlist"><span style="top:0.15000000000000002em;margin-right:0.07142857142857144em;margin-left:0em;"><span class="fontsize-ensurer reset-size5 size5"><span style="font-size:0em;">​</span></span><span class="reset-scriptstyle scriptscriptstyle cramped"><span class="mord scriptscriptstyle cramped"><span class="mord mathit">i</span><span class="mord mathit" style="margin-right:0.05724em;">j</span></span></span></span><span class="baseline-fix"><span class="fontsize-ensurer reset-size5 size5"><span style="font-size:0em;">​</span></span>​</span></span></span><span class="mbin">−</span><span class="mop"><span class="mop">min</span><span class="vlist"><span style="top:0.15em;margin-right:0.07142857142857144em;"><span class="fontsize-ensurer reset-size5 size5"><span style="font-size:0em;">​</span></span><span class="reset-scriptstyle scriptscriptstyle cramped"><span class="mord scriptscriptstyle cramped"><span class="mord mathrm">1</span><span class="mrel">≤</span><span class="mord mathit">t</span><span class="mrel">≤</span><span class="mord mathit">n</span></span></span></span><span class="baseline-fix"><span class="fontsize-ensurer reset-size5 size5"><span style="font-size:0em;">​</span></span>​</span></span></span><span class="mord"><span class="mord mathit">x</span><span class="vlist"><span style="top:0.15000000000000002em;margin-right:0.07142857142857144em;margin-left:0em;"><span class="fontsize-ensurer reset-size5 size5"><span style="font-size:0em;">​</span></span><span class="reset-scriptstyle scriptscriptstyle cramped"><span class="mord scriptscriptstyle cramped"><span class="mord mathit">t</span><span class="mord mathit" style="margin-right:0.05724em;">j</span></span></span></span><span class="baseline-fix"><span class="fontsize-ensurer reset-size5 size5"><span style="font-size:0em;">​</span></span>​</span></span></span></span></span></span><span class="baseline-fix"><span class="fontsize-ensurer reset-size5 size5"><span style="font-size:0em;">​</span></span>​</span></span></span><span class="sizing reset-size5 size5 reset-textstyle textstyle uncramped nulldelimiter"></span></span><span class="mpunct">,</span><span class="mord mathit">i</span><span class="mrel">=</span><span class="mord mathrm">1</span><span class="mpunct">,</span><span class="mord mathrm">2</span><span class="mpunct">,</span><span class="minner">⋯</span><span class="mpunct">,</span><span class="mord mathit">n</span><span class="mpunct">,</span><span class="mord mathit" style="margin-right:0.05724em;">j</span><span class="mrel">=</span><span class="mord mathrm">1</span><span class="mpunct">,</span><span class="mord mathrm">2</span><span class="mpunct">,</span><span class="minner">⋯</span><span class="mpunct">,</span><span class="mord mathit">m</span></span></span></span></p> <p>其中$R_j= \max_{1\le t \le n}x_{tj} - \min_{1\le t \le n}x_{tj} $</p> <p><strong>作用</strong>：变换后数据在[0,1]之间；极差为1，无量纲</p> <p><strong>对数变换</strong></p> <p><span class="katex"><span class="katex-mathml"><math><semantics><mrow><msubsup><mi>x</mi><mrow><mi>i</mi><mi>j</mi></mrow><mo>∗</mo></msubsup><mo>=</mo><mi>ln</mi><msub><mi>x</mi><mrow><mi>i</mi><mi>j</mi></mrow></msub><mo separator="true">,</mo><msub><mi>x</mi><mrow><mi>i</mi><mi>j</mi></mrow></msub><mo>&gt;</mo><mn>0</mn><mo separator="true">,</mo><mi>i</mi><mo>=</mo><mn>1</mn><mo separator="true">,</mo><mn>2</mn><mo separator="true">,</mo><mo>⋯</mo><mo separator="true">,</mo><mi>n</mi><mo separator="true">,</mo><mi>j</mi><mo>=</mo><mn>1</mn><mo separator="true">,</mo><mn>2</mn><mo separator="true">,</mo><mo>⋯</mo><mo separator="true">,</mo><mi>m</mi></mrow><annotation encoding="application/x-tex">x^*_{ij} = \ln x_{ij}, x_{ij} \gt 0,  i=1,2,\cdots, n, j = 1,2,\cdots, m</annotation></semantics></math></span><span aria-hidden="true" class="katex-html"><span class="strut" style="height:0.69444em;"></span><span class="strut bottom" style="height:1.077548em;vertical-align:-0.383108em;"></span><span class="base textstyle uncramped"><span class="mord"><span class="mord mathit">x</span><span class="vlist"><span style="top:0.24700000000000003em;margin-left:0em;margin-right:0.05em;"><span class="fontsize-ensurer reset-size5 size5"><span style="font-size:0em;">​</span></span><span class="reset-textstyle scriptstyle cramped"><span class="mord scriptstyle cramped"><span class="mord mathit">i</span><span class="mord mathit" style="margin-right:0.05724em;">j</span></span></span></span><span style="top:-0.363em;margin-right:0.05em;"><span class="fontsize-ensurer reset-size5 size5"><span style="font-size:0em;">​</span></span><span class="reset-textstyle scriptstyle uncramped"><span class="mord">∗</span></span></span><span class="baseline-fix"><span class="fontsize-ensurer reset-size5 size5"><span style="font-size:0em;">​</span></span>​</span></span></span><span class="mrel">=</span><span class="mop">ln</span><span class="mord"><span class="mord mathit">x</span><span class="vlist"><span style="top:0.15em;margin-right:0.05em;margin-left:0em;"><span class="fontsize-ensurer reset-size5 size5"><span style="font-size:0em;">​</span></span><span class="reset-textstyle scriptstyle cramped"><span class="mord scriptstyle cramped"><span class="mord mathit">i</span><span class="mord mathit" style="margin-right:0.05724em;">j</span></span></span></span><span class="baseline-fix"><span class="fontsize-ensurer reset-size5 size5"><span style="font-size:0em;">​</span></span>​</span></span></span><span class="mpunct">,</span><span class="mord"><span class="mord mathit">x</span><span class="vlist"><span style="top:0.15em;margin-right:0.05em;margin-left:0em;"><span class="fontsize-ensurer reset-size5 size5"><span style="font-size:0em;">​</span></span><span class="reset-textstyle scriptstyle cramped"><span class="mord scriptstyle cramped"><span class="mord mathit">i</span><span class="mord mathit" style="margin-right:0.05724em;">j</span></span></span></span><span class="baseline-fix"><span class="fontsize-ensurer reset-size5 size5"><span style="font-size:0em;">​</span></span>​</span></span></span><span class="mrel">&gt;</span><span class="mord mathrm">0</span><span class="mpunct">,</span><span class="mord mathit">i</span><span class="mrel">=</span><span class="mord mathrm">1</span><span class="mpunct">,</span><span class="mord mathrm">2</span><span class="mpunct">,</span><span class="minner">⋯</span><span class="mpunct">,</span><span class="mord mathit">n</span><span class="mpunct">,</span><span class="mord mathit" style="margin-right:0.05724em;">j</span><span class="mrel">=</span><span class="mord mathrm">1</span><span class="mpunct">,</span><span class="mord mathrm">2</span><span class="mpunct">,</span><span class="minner">⋯</span><span class="mpunct">,</span><span class="mord mathit">m</span></span></span></span></p> <p><strong>作用</strong>：将具有指数特征的数据结构变换为现行数据结构</p> <h2 id="大数据库"><a href="#大数据库" class="header-anchor">#</a> 大数据库</h2> <h3 id="_1-理解nosql与newsql"><a href="#_1-理解nosql与newsql" class="header-anchor">#</a> （1）理解NoSQL与NewSQL；</h3> <p>NoSQL = Not-Only-SQL</p> <ol><li><strong>不需要预定义模式</strong>：数据中的每条记录都可能有不同的属性和格式。</li> <li><strong>无共享架构</strong>：相对于将所有数据存储到服务中，达到数据共享。存在 的问题是访问数据需要带宽。NoSQL将数据划分后存储在各个本地服务器 上。</li> <li><strong>弹性可扩展</strong>：可以在系统运行的时候，动态增加或者删除节点。不需 要停机维护，数据可以自动迁移。</li> <li><strong>分区</strong>：NoSQL数据库需要将数据进行分区，将记录分散在多个节点 上面。这样既提高了并行性能，又能保证没有单点失效的问题。</li> <li><strong>异步复制</strong>：和RAID存储系统不同的是，NoSQL中的复制，往往是基于日志的异步复制。优点是不会因网络传输引起迟延。缺点是并不总是能保 证一致性，这样的方式在出现故障的时候，可能会丢失少量的数据。</li></ol> <p>NoSQL分类如下</p> <p><img src="/assets/img/image-20200626162246437.b2105a25.png" alt="image-20200626162246437"></p> <blockquote><p>NewSQL是对各种新的可扩展/高性能数据库的简称，这类数据库不仅具有NoSQL对海量数据的存储管理能力，还保持了传统数据库支持ACID和SQL等特性。</p></blockquote> <h3 id="_2-理解批量处理技术"><a href="#_2-理解批量处理技术" class="header-anchor">#</a> （2）理解批量处理技术；</h3> <p>批处理技术包含以下特征：</p> <p>有界：批处理数据集代表数据的有限集合
持久：数据通常始终存储在某种类型的持久存储位置中
大量：批处理操作通常是处理极为海量数据集的唯一方法</p> <blockquote><p>需要处理大量数据的任务通常最适合用批处理操作进行处理。无论直接从持久存储设备处理数据集，或首先将数据集载入内存，批处理系统在设计过程中就充分考虑了数据的量，可提供充足的处理资源。由于批处理在应对大量持久数据方面的表现极为出色，因此经常被用于对历史数据进行分析。</p> <p>大量数据的处理需要付出大量时间，因此批处理不适合对处理时间要求较高的场合。</p></blockquote> <p>hadoop的MapReduce引擎提供了批处理技术</p> <h3 id="_3-了解流计算原理"><a href="#_3-了解流计算原理" class="header-anchor">#</a> （3）了解流计算原理；</h3> <p>流处理不对整个数据集处理，而是对系统传输的每个数据项执行操作</p> <blockquote><p>完整数据集只能代表截至目前已经进入到系统中的数据总量。</p> <p>工作数据集在特定时间只能代表某个单一数据项。</p> <p>处理工作是基于事件的，除非明确停止否则没有“尽头”。</p> <p>处理结果立刻可用，并会随着新数据的抵达继续更新。</p></blockquote> <p>Apache Samza是一种流处理框架</p> <h3 id="_4-了解图计算原理"><a href="#_4-了解图计算原理" class="header-anchor">#</a> （4）了解图计算原理；</h3> <blockquote><p>Pregel作为分布式图计算的计算框架,主要用于图遍历、最短路径、PageRank计算等等。</p></blockquote> <p>图计算以有向图作为输入，包含顶点和边，顶点有String类型的ID，顶点和有向边都有一个用户定义的值与之关联</p> <blockquote><p>Pregel的计算过程是由一系列被称为“超步”的迭代组成的。</p> <p>在每个超步中，每个顶点上面都会并行执行用户自定义的函数，该函数描述了一个顶点V在一个超步S中需要执行的操作。</p> <p>该函数可以读取前一个超步(S-1)中其他顶点发送给顶点V的消息，执行相应计算后，修改顶点V及其出射边的状态，然后沿着顶点V的出射边发送消息给其他顶点，而且，一个消息可能经过多条边的传递后被发送到任意已知ID的目标顶点上去。</p> <p>这些消息将会在下一个超步(S+1)中被目标顶点接收，然后像上述过程一样开始下一个超步(S+1)的迭代过程。</p> <h2 id="在pregel计算过程中-一个算法什么时候可以结束-是由所有顶点的状态决定的。"><a href="#在pregel计算过程中-一个算法什么时候可以结束-是由所有顶点的状态决定的。" class="header-anchor">#</a> 在Pregel计算过程中，一个算法什么时候可以结束，是由所有顶点的状态决定的。</h2> <p>版权声明：本文为CSDN博主「曹世宏的博客」的原创文章，遵循CC 4.0 BY-SA版权协议，转载请附上原文出处链接及本声明。
原文链接：https://blog.csdn.net/qq_38265137/article/details/80547763</p></blockquote> <h3 id="_5-了解查询分析计算原理"><a href="#_5-了解查询分析计算原理" class="header-anchor">#</a> （5）了解查询分析计算原理；</h3> <p>hadoop的hive实现了查询分析计算。对数据仓库的数据，使用类似sql语句的方法进行查询分析</p> <h3 id="_6-掌握hbase模型"><a href="#_6-掌握hbase模型" class="header-anchor">#</a> （6）掌握HBASE模型</h3> <p><img src="/assets/img/image-20200626163444876.922be037.png" alt="image-20200626163444876"></p> <p>HBase中需要根据行键、列族、列限定符和时间戳来确定一个单元格， 因此，可以视为一个“四维坐标”，即[行键, 列族, 列限定符, 时间戳]</p> <p>特点</p> <ol><li>数据类型：HBase 则把数据存储为未经解释的字符串</li> <li>数据操作：操作则只有简单的插入、查询、 删除、清空等</li> <li>存储模式：HBase 是基于列存储的，每个列族都由几个文件保存，不同列族的 文件是分离的；</li> <li>数据索引：HBase只有一个索引— —行键</li> <li>数据维护：HBase中执行更新操作时，并不会删除数据旧的版本，而是 生成一个新的版本，旧有的版本仍然保留。</li> <li>可伸缩性：HBase就是为了实现灵活的水平 扩展而开发的，能够轻易地在集群中实现性能的伸缩。</li></ol> <p>hbase系统架构</p> <p><img src="/assets/img/image-20200626165616344.0722a337.png" alt="image-20200626165616344"></p> <h2 id="大数据分析"><a href="#大数据分析" class="header-anchor">#</a> 大数据分析</h2> <h3 id="_1-了解数据分析的原则、特点"><a href="#_1-了解数据分析的原则、特点" class="header-anchor">#</a> （1）了解数据分析的原则、特点；</h3> <p>原则</p> <ol><li>分析需要有明确的目标</li> <li>需要针对可能的原因追踪分析</li> <li>需要用测试数据验证模型的准确性</li></ol> <p>特点</p> <ol><li>大数据分析是可视化分析</li> <li>理论核心是数据挖掘算法</li> <li>重要应用是预测性分析</li> <li>广泛用于网络数据挖掘</li> <li>重视数据质量和数据管理</li></ol> <h3 id="_2-掌握大数据分析流程"><a href="#_2-掌握大数据分析流程" class="header-anchor">#</a> （2）掌握大数据分析流程；</h3> <p>业务理解-&gt;数据认知-&gt;特征工程-&gt;数据建模-&gt;评估与解释</p> <h3 id="_3-理解大数据分析技术、难点"><a href="#_3-理解大数据分析技术、难点" class="header-anchor">#</a> （3）理解大数据分析技术、难点 ；</h3> <p><img src="/assets/img/image-20200626174354641.aac82553.png" alt="image-20200626174354641"></p> <p>难点：</p> <ul><li>可扩展性</li> <li>可用性</li> <li>领域知识的结合</li> <li>结果的检验</li></ul> <h3 id="_4-理解数据认知的基本方法"><a href="#_4-理解数据认知的基本方法" class="header-anchor">#</a> （4）理解数据认知的基本方法；</h3> <p>数据预处理</p> <p>概率分析：<span class="katex"><span class="katex-mathml"><math><semantics><mrow><mi>μ</mi></mrow><annotation encoding="application/x-tex">\mu</annotation></semantics></math></span><span aria-hidden="true" class="katex-html"><span class="strut" style="height:0.43056em;"></span><span class="strut bottom" style="height:0.625em;vertical-align:-0.19444em;"></span><span class="base textstyle uncramped"><span class="mord mathit">μ</span></span></span></span>检验，t检验，F检验，卡方检验</p> <p><img src="/assets/img/image-20200626175112081.3a04955c.png" alt="image-20200626175112081"></p> <p>对比分析：同比、环比、定基比</p> <p>细分分析</p> <p>交叉分析</p> <p>相关分析</p> <h3 id="_5-掌握特征工程的作用和基本方法"><a href="#_5-掌握特征工程的作用和基本方法" class="header-anchor">#</a> （5）掌握特征工程的作用和基本方法；</h3> <p>作用：设计指标或提取关键特征来代表整个对象</p> <p>基本方法：</p> <ol><li>PCA(主成分分析)</li> <li>FA（因子分析）</li></ol> <p>指标设计技巧</p> <ol><li>时间戳处理</li> <li>分解类别变量</li> <li>分箱/分区</li> <li>交叉特征</li></ol> <h3 id="_6-掌握数据建模的常用方法"><a href="#_6-掌握数据建模的常用方法" class="header-anchor">#</a> （6）掌握数据建模的常用方法</h3> <ul><li>决策树</li> <li>关联分析</li> <li>回归分析</li> <li>聚类分析</li></ul> <h3 id="_7-了解通用计算引擎spark"><a href="#_7-了解通用计算引擎spark" class="header-anchor">#</a> （7）了解通用计算引擎Spark</h3> <p>Spark使用Scala语言进行实现</p> <p>(1)运行速度快 (2)易用性好 (3)通用性强 (4)支持多种数据源</p> <p><img src="/assets/img/image-20200626181756267.e3335a06.png" alt="image-20200626181756267"></p> <p>Spark把中间数据放到内存中，迭代运算效率高。</p> <p>Spark 容错性高</p> <p>Spark提供的数据集操作类型有很多种。大致分为 ： Transformations和Actions两大类</p> <p>需要反复操作的次数越多，所需读取的数据量越大，受益越大，</p> <p>不适合增量修改的应用模型</p> <p>满足数据量不是特别大的实时统计分析需求</p> <h3 id="_8-掌握hive的数据分析方法"><a href="#_8-掌握hive的数据分析方法" class="header-anchor">#</a> （8）掌握hive的数据分析方法；</h3> <blockquote><p>Hive是建立在 Hadoop 上的数据仓库基础构架。它提供了一系列的工具， 可以用来进行数据提取转化加载（ETL），这是一种可以存储、查询和分析 存储在 Hadoop 中的大规模数据的机制。Hive 定义了简单的类 SQL 查询语 言，称为 HQL。</p> <p>支持索引，加快数据查询。</p> <p>支持不同的存储类型，例如，纯文本文件、HBase中的文件。 将元数据保存在关系数据库中，大大减少了在查询过程中执行语义检查 的时间。</p> <p>可以直接使用存储在Hadoop文件系统中的数据。</p> <p>内置大量用户函数UDF来操作时间、字符串和其他的数据挖掘工具，支 持用户扩展UDF 函数来完成内置函数无法实现的操作。</p> <p>类SQL 的查询方式，将SQL 查询转换为MapReduce 的job在Hadoop集群 上执行。</p></blockquote> <p><img src="/assets/img/image-20200626182444718.d85e8dea.png" alt="image-20200626182444718"></p> <h2 id="大数据可视化技术"><a href="#大数据可视化技术" class="header-anchor">#</a> 大数据可视化技术</h2> <h3 id="_1-了解数据可视化概念和作用"><a href="#_1-了解数据可视化概念和作用" class="header-anchor">#</a> （1）了解数据可视化概念和作用</h3> <p><strong>概念</strong>：数据可视化主要是借助于图形化手段，清晰有效地传达信息</p> <p><strong>作用</strong>：</p> <ol><li>帮助人类消化大量信息</li> <li>清晰展现数据背后的意义</li> <li>辅助决策</li> <li>给人强大的视觉冲击</li></ol> <h3 id="_2-理解可视化建议"><a href="#_2-理解可视化建议" class="header-anchor">#</a> （2）理解可视化建议；</h3> <p><img src="/assets/img/image-20200626183432021.e486e109.png" alt="image-20200626183432021"></p> <p><img src="/assets/img/image-20200626183531113.6460d08b.png" alt="image-20200626183531113"></p> <h3 id="_3-了解图表的作用"><a href="#_3-了解图表的作用" class="header-anchor">#</a> （3）了解图表的作用；</h3> <p><strong>直方图</strong>：表示定量变量在各个区间的分布规律</p> <p><img src="/assets/img/image-20200626184308954.303acb0a.png" alt="image-20200626184308954"></p> <p><strong>饼图</strong>：表示一个定性变量的各个取值的比例</p> <p><img src="/assets/img/image-20200626184317789.17007f2c.png" alt="image-20200626184317789"></p> <p><strong>柱状图</strong>：表示两个定性变量的各个取值的数量对比</p> <p><img src="/assets/img/image-20200626184327288.d1a4b535.png" alt="image-20200626184327288"></p> <p><strong>堆积柱状图</strong>：表示一个定性变量A的各个取值的数量，同时表示定性变量B的取值在每个A的取值中的比例</p> <p><img src="/assets/img/image-20200626184340462.1b565a13.png" alt="image-20200626184340462"></p> <p><strong>折线图</strong>：表示一个定量变量在时间上的变化</p> <p><img src="/assets/img/image-20200626184348756.2fe9e926.png" alt="image-20200626184348756"></p> <p><strong>散点图</strong>：表示两个定量变量的关系</p> <p><img src="/assets/img/image-20200626184358618.e9efa0a8.png" alt="image-20200626184358618"></p> <p><strong>气泡图</strong>：在散点图的基础上用气泡的大小表示第三个定量变量的取值，从而表示三个变量的关系</p> <p><img src="/assets/img/image-20200626184406992.32694166.png" alt="image-20200626184406992"></p> <p><strong>风玫瑰图</strong>：花瓣的大小表示定性变量A的数量，在一个花瓣上的每个分层表示定性变量B在A的某个取值的比例</p> <p><img src="/assets/img/image-20200626184428313.bbe15f12.png" alt="image-20200626184428313"></p> <p><strong>相关系数图</strong>：表示多个定量变量的线性相关关系。</p> <p><img src="/assets/img/image-20200626184438943.e60b4281.png" alt="image-20200626184438943"></p> <p><strong>箱形图</strong>：表示定量变量的五个分位数（min,max, 25%,median, 75%),描述变量的分布，用于发现异常值</p> <p><img src="/assets/img/image-20200626184450178.3a81e2e6.png" alt="image-20200626184450178"></p> <p><strong>雷达图</strong>：表示若干个记录在多个属性上的对比</p> <p><img src="/assets/img/image-20200626184504064.5ba1a331.png" alt="image-20200626184504064"></p> <p><strong>词云图</strong>：突出表示定性变量的高频取值</p> <p><img src="/assets/img/image-20200626184514110.c0f0631b.png" alt="image-20200626184514110"></p> <p><strong>漏斗图</strong>：表示一个定性变量的数量从多到少的一个排列</p> <p><img src="/assets/img/image-20200626184536580.57c073ef.png" alt="image-20200626184536580"></p> <h3 id="_4-熟练掌握基本统计的绘制方法"><a href="#_4-熟练掌握基本统计的绘制方法" class="header-anchor">#</a> （4）熟练掌握基本统计的绘制方法；</h3> <p>可选工具有：Excel， SPSS, Tableau， R语言</p> <p>绘制方法略（以后再专门写一篇用excel绘制统计图的文章）</p> <h3 id="_5-理解ggplot2图层绘图的基本思想"><a href="#_5-理解ggplot2图层绘图的基本思想" class="header-anchor">#</a> （5）理解ggplot2图层绘图的基本思想；</h3> <p>ggplot2作图过程中最重要的三个概念：映射、图层和几何对象</p> <p>映射将数据集的属性关联到图形属性上</p> <p>图层是图形绘制的区域，多个图层可以穿透叠加在同一个画布上</p> <p>几何对象执行图形的实际渲染，控制生成的图标类型</p> <h2 id="大数据应用"><a href="#大数据应用" class="header-anchor">#</a> 大数据应用</h2> <h3 id="_1-理解医疗大数据分析过程"><a href="#_1-理解医疗大数据分析过程" class="header-anchor">#</a> （1）理解医疗大数据分析过程；</h3> <ol><li>移动医疗</li> <li>基因测序</li></ol> <h3 id="_2-理解交通大数据分析过程"><a href="#_2-理解交通大数据分析过程" class="header-anchor">#</a> （2）理解交通大数据分析过程；</h3> <ol><li><p>收集交通大数据</p></li> <li><p>定义拥堵程度</p></li> <li><p>绘制拥堵地图</p></li> <li><p>分析城市拥堵模式</p></li> <li><p>分析交通政策实施前后的变化</p></li> <li><p>分析天气、路面、阳光等对交通事故的影响</p></li> <li><p>预测航班起飞时间</p></li> <li><p>购物篮分析</p></li> <li><p>物流分析</p></li></ol> <h2 id="大数据安全"><a href="#大数据安全" class="header-anchor">#</a> 大数据安全</h2> <h3 id="_1-了解大数据安全的意义和面临的挑战"><a href="#_1-了解大数据安全的意义和面临的挑战" class="header-anchor">#</a> （1）了解大数据安全的意义和面临的挑战；</h3> <ol><li>大数据成为网络攻击的显著目标</li> <li>大数据加大隐私泄露风险</li> <li>大数据对现有的存储和安防措施提出挑战</li> <li>大数据技术被应用到攻击手段</li> <li>大数据成为高级可持续攻击的载体</li></ol> <h3 id="_2-理解安全威胁的种类和特征"><a href="#_2-理解安全威胁的种类和特征" class="header-anchor">#</a> （2）理解安全威胁的种类和特征；</h3> <ol><li>大数据基础设施安全威胁</li> <li>大数据存储安全威胁</li> <li>大数据的隐私泄露</li> <li>网络化社会使大数据极易成为攻击目标</li> <li>过度相信和沉迷于数据</li></ol> <h3 id="_3-了解大数据与网络攻击监测"><a href="#_3-了解大数据与网络攻击监测" class="header-anchor">#</a> （3）了解大数据与网络攻击监测；</h3> <p>大数据时代下传统安全手段难以满足业务需要</p> <p>传统安全应急响应，无法解决大数据时代下面临的察觉晚、数据已损失的问题</p> <h3 id="_4-掌握大数据安全分析的基本方法"><a href="#_4-掌握大数据安全分析的基本方法" class="header-anchor">#</a> （4）掌握大数据安全分析的基本方法；</h3> <p><img src="/assets/img/image-20200626192549507.6b40eb0f.png" alt="image-20200626192549507"></p> <p>基本方法</p> <ol><li>检测威胁：实时检测来自内部与外部的攻击</li> <li>事件还原：通过诸多数据来源，还原安全事件真相</li> <li>事件追溯：解读攻击者 的真实身份、 背景、动机</li> <li>漏洞分析：出具漏洞修补方案，抵 御二次攻击</li></ol> <h3 id="_5-了解大数据安全标准。"><a href="#_5-了解大数据安全标准。" class="header-anchor">#</a> （5）了解大数据安全标准。</h3> <p><img src="/assets/img/image-20200626200506015.12ed9a4b.png" alt="image-20200626200506015"></p> <h2 id="参考文献"><a href="#参考文献" class="header-anchor">#</a> 参考文献</h2> <p><a href="https://blog.csdn.net/weixin_43093501/article/details/95225833" target="_blank" rel="noopener noreferrer">大数据的流处理和批处理及其框架<span><svg xmlns="http://www.w3.org/2000/svg" aria-hidden="true" focusable="false" x="0px" y="0px" viewBox="0 0 100 100" width="15" height="15" class="icon outbound"><path fill="currentColor" d="M18.8,85.1h56l0,0c2.2,0,4-1.8,4-4v-32h-8v28h-48v-48h28v-8h-32l0,0c-2.2,0-4,1.8-4,4v56C14.8,83.3,16.6,85.1,18.8,85.1z"></path> <polygon fill="currentColor" points="45.7,48.7 51.3,54.3 77.2,28.5 77.2,37.2 85.2,37.2 85.2,14.9 62.8,14.9 62.8,22.9 71.5,22.9"></polygon></svg> <span class="sr-only">(opens new window)</span></span></a></p></div> <footer class="page-edit"><!----> <!----> <a rel="license" href="https://creativecommons.org/licenses/by-sa/4.0/deed.zh"><img alt="知识共享许可协议" src="" style="border-width:0"></a><br>本作品采用<a rel="license" href="http://creativecommons.org/licenses/by/4.0/">知识共享署名 4.0 国际许可协议</a>进行许可。

   
</footer> <div class="page-nav"><p class="inner"><span class="prev">
      ←
      <a href="/知识点总结/人工智能总结.html" class="prev">
        人工智能总结
      </a></span> <span class="next"><a href="/知识点总结/常用快捷键总结.html">
        第一篇博客
      </a>
      →
    </span></p></div>  <footer style="text-align:center;"><a href="https://beian.miit.gov.cn">
    粤ICP备2021020303号</a></footer></main></div><div class="global-ui"><!----></div></div>
    <script src="/assets/js/app.54ea76a2.js" defer></script><script src="/assets/js/2.c554ece5.js" defer></script><script src="/assets/js/3.9d562ae1.js" defer></script>
  </body>
</html>
